[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505913074 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003111] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.012014] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.015024] *** VALIDATE selinux *** [ 0.023725] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027618] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.031369] *** VALIDATE proc *** [ 0.032181] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034216] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036078] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041476] debug: unmapping init [mem 0xffffffffb7859000-0xffffffffb7860fff] [ 0.043132] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044655] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.045981] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049019] ... fixed-purpose events: 3 [ 0.050019] ... event mask: 000000070000000f [ 0.051318] rcu: Hierarchical SRCU implementation. [ 0.053358] smp: Bringing up secondary CPUs ... [ 0.054606] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.058399] smp: Brought up 1 node, 4 CPUs [ 0.060016] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140017] node 0 deferred pages initialised in 76ms [ 0.142191] devtmpfs: initialized [ 0.143177] x86/mm: Memory block size: 128MB [ 0.145442] gcov: version magic: 0x41383552 [ 0.148283] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150000] pinctrl core: initialized pinctrl subsystem [ 0.151189] [ 0.151705] ************************************************************* [ 0.154014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156007] ** ** [ 0.157007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158009] ** ** [ 0.160008] ** This means that this kernel is built to expose internal ** [ 0.161010] ** IOMMU data structures, which may compromise security on ** [ 0.162009] ** your system. ** [ 0.163008] ** ** [ 0.164009] ** If you see this message and you are not debugging the ** [ 0.166009] ** kernel, report this immediately to your vendor! ** [ 0.167007] ** ** [ 0.169009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171010] ************************************************************* [ 0.172635] NET: Registered protocol family 16 [ 0.174348] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181040] cpuidle: using governor menu [ 0.182612] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183419] PCI: Using configuration type 1 for base access [ 0.184096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197098] cryptd: max_cpu_qlen set to 1000 [ 0.199190] ACPI: Added _OSI(Module Device) [ 0.200010] ACPI: Added _OSI(Processor Device) [ 0.200965] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207915] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214508] ACPI: Interpreter enabled [ 0.216065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218018] ACPI: Using IOAPIC for interrupt routing [ 0.219113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222355] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229888] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240201] acpiphp: Slot [2] registered [ 0.241108] acpiphp: Slot [5] registered [ 0.242128] acpiphp: Slot [6] registered [ 0.244217] acpiphp: Slot [7] registered [ 0.245117] acpiphp: Slot [8] registered [ 0.247194] acpiphp: Slot [9] registered [ 0.249216] acpiphp: Slot [10] registered [ 0.250206] acpiphp: Slot [3] registered [ 0.252151] acpiphp: Slot [4] registered [ 0.254173] acpiphp: Slot [11] registered [ 0.255128] acpiphp: Slot [12] registered [ 0.257132] acpiphp: Slot [13] registered [ 0.259137] acpiphp: Slot [14] registered [ 0.260150] acpiphp: Slot [15] registered [ 0.262159] acpiphp: Slot [16] registered [ 0.264159] acpiphp: Slot [17] registered [ 0.265116] acpiphp: Slot [18] registered [ 0.267148] acpiphp: Slot [19] registered [ 0.269162] acpiphp: Slot [20] registered [ 0.270155] acpiphp: Slot [21] registered [ 0.272140] acpiphp: Slot [22] registered [ 0.274160] acpiphp: Slot [23] registered [ 0.276176] acpiphp: Slot [24] registered [ 0.278141] acpiphp: Slot [25] registered [ 0.279207] acpiphp: Slot [26] registered [ 0.281179] acpiphp: Slot [27] registered [ 0.283164] acpiphp: Slot [28] registered [ 0.284140] acpiphp: Slot [29] registered [ 0.286200] acpiphp: Slot [30] registered [ 0.287136] acpiphp: Slot [31] registered [ 0.289069] PCI host bridge to bus 0000:00 [ 0.290026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309812] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321735] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.324017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.329721] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332709] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336962] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368030] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.391022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419119] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.445032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456698] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.469030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.477019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.503806] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.515029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.554328] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.562026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.569016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.595920] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.602022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.635515] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.638390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.640493] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.642385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.644191] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648073] iommu: Default domain type: Passthrough [ 0.651070] SCSI subsystem initialized [ 0.653173] ACPI: bus type USB registered [ 0.655133] usbcore: registered new interface driver usbfs [ 0.656098] usbcore: registered new interface driver hub [ 0.658085] usbcore: registered new device driver usb [ 0.660287] pps_core: LinuxPPS API ver. 1 registered [ 0.662016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665066] PTP clock support registered [ 0.668049] EDAC MC: Ver: 3.0.0 [ 0.669481] PCI: Using ACPI for IRQ routing [ 0.673151] NetLabel: Initializing [ 0.674011] NetLabel: domain hash size = 128 [ 0.675011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.677096] NetLabel: unlabeled traffic allowed by default [ 0.680168] vgaarb: loaded [ 0.681298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.683022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688435] clocksource: Switched to clocksource kvm-clock [ 0.778778] VFS: Disk quotas dquot_6.6.0 [ 0.779937] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.781772] *** VALIDATE ramfs *** [ 0.782659] *** VALIDATE hugetlbfs *** [ 0.783757] pnp: PnP ACPI init [ 0.786255] pnp: PnP ACPI: found 6 devices [ 0.800302] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.802889] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.805318] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.807741] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.810292] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.812981] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.816175] NET: Registered protocol family 2 [ 0.818693] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.823105] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.826575] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.832422] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.836185] TCP: Hash tables configured (established 65536 bind 65536) [ 0.839195] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.842504] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.845566] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.848617] NET: Registered protocol family 1 [ 0.851571] RPC: Registered named UNIX socket transport module. [ 0.853605] RPC: Registered udp transport module. [ 0.855542] RPC: Registered tcp transport module. [ 0.857333] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.859521] NET: Registered protocol family 44 [ 0.860781] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.862397] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.863856] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.866261] PCI: CLS 0 bytes, default 64 [ 0.867852] Unpacking initramfs... [ 2.231231] debug: unmapping init [mem 0xffff91bb7cc54000-0xffff91bb7ffbffff] [ 2.233638] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.235325] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.237208] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.660932] Initialise system trusted keyrings [ 2.662018] Key type blacklist registered [ 2.663248] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.672979] zbud: loaded [ 2.675425] *** VALIDATE nfs *** [ 2.676257] *** VALIDATE nfs4 *** [ 2.677205] pstore: using deflate compression [ 2.680467] Platform Keyring initialized [ 2.778087] NET: Registered protocol family 38 [ 2.779533] Key type asymmetric registered [ 2.780644] Asymmetric key parser 'x509' registered [ 2.782432] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.784567] io scheduler mq-deadline registered [ 2.785920] io scheduler kyber registered [ 2.787411] io scheduler bfq registered [ 2.789189] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.791413] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.793927] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.796835] ACPI: Power Button [PWRF] [ 2.802575] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.808630] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.818206] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.823311] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.841703] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.871127] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.900925] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.906128] Non-volatile memory driver v1.3 [ 2.907847] Linux agpgart interface v0.103 [ 2.940845] virtio_blk virtio1: [vda] 133936 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.943265] vda: detected capacity change from 0 to 68575232 [ 2.958744] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.961463] vdb: detected capacity change from 0 to 1073741824 [ 2.975546] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.978115] vdc: detected capacity change from 0 to 2621440000 [ 2.996101] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.998131] vdd: detected capacity change from 0 to 2621440000 [ 3.009977] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.011952] vde: detected capacity change from 0 to 4294967296 [ 3.024120] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.026119] vdf: detected capacity change from 0 to 4294967296 [ 3.031341] libphy: Fixed MDIO Bus: probed [ 3.038629] usbcore: registered new interface driver usbserial_generic [ 3.041093] usbserial: USB Serial support registered for generic [ 3.043442] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.047705] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.049676] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.052302] mousedev: PS/2 mouse device common for all mice [ 3.055686] rtc_cmos 00:05: RTC can wake from S4 [ 3.058820] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.059485] rtc_cmos 00:05: registered as rtc0 [ 3.064414] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.068243] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.068548] intel_pstate: CPU model not supported [ 3.074438] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.079079] hid: raw HID events driver (C) Jiri Kosina [ 3.081054] usbcore: registered new interface driver usbhid [ 3.083294] usbhid: USB HID core driver [ 3.085116] drop_monitor: Initializing network drop monitor service [ 3.087732] Initializing XFRM netlink socket [ 3.090115] NET: Registered protocol family 10 [ 3.093384] Segment Routing with IPv6 [ 3.094384] NET: Registered protocol family 17 [ 3.095744] mpls_gso: MPLS GSO support [ 3.102929] RAS: Correctable Errors collector initialized. [ 3.104181] AVX version of gcm_enc/dec engaged. [ 3.105284] AES CTR mode by8 optimization enabled [ 3.179593] sched_clock: Marking stable (3179563059, 0)->(4000086609, -820523550) [ 3.183914] registered taskstats version 1 [ 3.185744] Loading compiled-in X.509 certificates [ 3.187190] zswap: loaded using pool lzo/zbud [ 3.214345] Key type big_key registered [ 3.225849] Key type encrypted registered [ 3.227173] ima: No TPM chip found, activating TPM-bypass! [ 3.228508] ima: Allocated hash algorithm: sha1 [ 3.230009] ima: No architecture policies found [ 3.231259] evm: Initialising EVM extended attributes: [ 3.232628] evm: security.selinux [ 3.233830] evm: security.ima [ 3.234605] evm: security.capability [ 3.235495] evm: HMAC attrs: 0x1 [ 3.237781] rtc_cmos 00:05: setting system clock to 2026-04-03 11:33:49 UTC (1775216029) [ 3.243502] debug: unmapping init [mem 0xffffffffb8803000-0xffffffffb89fffff] [ 3.245798] debug: unmapping init [mem 0xffffffffb7582000-0xffffffffb7858fff] [ 3.254131] Write protecting the kernel read-only data: 28672k [ 3.257269] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.259671] debug: unmapping init [mem 0xffffffffb6514000-0xffffffffb65fffff] [ 3.290292] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.296788] systemd[1]: Detected virtualization kvm. [ 3.298563] systemd[1]: Detected architecture x86-64. [ 3.300019] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.323878] systemd[1]: No hostname configured. [ 3.325193] systemd[1]: Set hostname to . [ 3.326802] random: systemd: uninitialized urandom read (16 bytes read) [ 3.328624] systemd[1]: Initializing machine ID from random generator. [ 3.376188] random: ln: uninitialized urandom read (6 bytes read) [ 3.462497] random: systemd: uninitialized urandom read (16 bytes read) [ 3.465734] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.470530] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.474515] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. [ 3.986127] random: fast init done Starting dracut pre-udev hook... [ 4.054589] device-mapper: uevent: version 1.0.3 [ 4.056500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.815071] virtio_net virtio0 ens2: renamed from eth0 [ 4.859214] scsi host0: ata_piix [ 4.873690] scsi host1: ata_piix [ 4.875384] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.877825] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.707058] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.740433] random: crng init done [ 9.741890] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.249254] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.519498] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.792141] SELinux: Disabled at runtime. [ 11.854614] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.862614] systemd[1]: Detected virtualization kvm. [ 11.864469] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.372834] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.376387] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.380881] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.385151] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.389260] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.397390] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.405489] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ 12.561281] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.869897] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.183389] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.220340] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.483214] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.499220] EDAC sbridge: Ver: 1.1.2 [ 15.009331] Key type dns_resolver registered [ 15.330098] NFS: Registering the id_resolver key type [ 15.332222] Key type id_resolver registered [ 15.333647] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg109-server login: [ 30.969554] hrtimer: interrupt took 5542811 ns [ 69.164886] libcfs: loading out-of-tree module taints kernel. [ 69.199895] Key type ._llcrypt registered [ 69.201637] Key type .llcrypt registered [ 69.325519] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 88.773480] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 90.137276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.169595] alg: No test for adler32 (adler32-zlib) [ 91.902242] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 92.906971] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 94.815604] Key type lgssc registered [ 96.655302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.874175] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.640854] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 169.020946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.044745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.337841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.388847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.537268] Lustre: lustre-MDT0000: new disk, initializing [ 170.704846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.741414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.191406] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.245498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.346805] Lustre: 6504:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.383866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.395876] Lustre: Skipped 1 previous similar message [ 189.553607] Lustre: lustre-MDT0001: new disk, initializing [ 189.669778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.714077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 189.723120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.636750] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.872945] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.737963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.031665] Lustre: lustre-OST0000: new disk, initializing [ 210.037568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.131576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.874308] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.661397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.676598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.739923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.694261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.861247] Lustre: lustre-OST0001: new disk, initializing [ 230.867183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 230.969251] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.441412] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.142452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.150949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.199180] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.929033] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.798329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.605183] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.955135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing yml_node [ 274.483356] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 277.142834] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 280.213650] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 281.856169] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 3 07:38:26 EDT 2026 [ 302.026871] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 303.415369] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.967938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.969197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.981968] Lustre: Skipped 1 previous similar message [ 319.990873] Lustre: Skipped 3 previous similar messages [ 325.109901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.113486] Lustre: Skipped 3 previous similar messages [ 330.210656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.279615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 333.514451] Lustre: server umount lustre-MDT0000 complete [ 337.321508] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216363 with bad export cookie 7158135307827157385 [ 337.332812] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.337895] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.447805] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.448263] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.448623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.448627] Lustre: Skipped 3 previous similar messages [ 340.499537] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 340.550065] Lustre: Skipped 3 previous similar messages [ 342.496263] LustreError: 8851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.531473] LustreError: 8851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 344.180156] Lustre: server umount lustre-MDT0001 complete [ 349.089292] Lustre: server umount lustre-OST0000 complete [ 352.968136] Lustre: server umount lustre-OST0001 complete [ 361.016456] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 370.859853] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 406.522956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.863975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.917719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.034979] Lustre: lustre-MDT0000: new disk, initializing [ 407.127395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.148204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.704664] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 423.057429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.147995] Lustre: 14981:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.193850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.195742] Lustre: Skipped 1 previous similar message [ 423.239652] Lustre: lustre-MDT0001: new disk, initializing [ 423.312114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.363219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.378334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.945556] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 435.361678] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 439.351308] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 445.270481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 445.592883] Lustre: lustre-OST0000: new disk, initializing [ 445.601015] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 445.712666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.025366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 447.029881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 447.196700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.530509] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 458.825391] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 463.377431] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 463.569519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 467.849218] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 468.041414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.056884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.056930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.072938] LustreError: Skipped 1 previous similar message [ 473.075761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.075769] Lustre: Skipped 3 previous similar messages [ 473.105305] Lustre: Skipped 1 previous similar message [ 475.951731] Lustre: server umount lustre-OST0000 complete [ 479.716662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 479.727724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.298228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 483.302696] Lustre: Skipped 2 previous similar messages [ 486.043064] Lustre: server umount lustre-MDT0000 complete [ 488.417967] LustreError: 14991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.467208] LustreError: 14991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 489.357183] LustreError: 14975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216515 with bad export cookie 7158135307827158918 [ 489.360163] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.370517] LustreError: 14975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 489.674473] Lustre: server umount lustre-MDT0001 complete [ 494.827729] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 07:41:59 (1775216519) [ 502.310112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.911014] LustreError: 18844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.003453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.005563] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 508.390601] LustreError: 18845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.946848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.238052] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 524.365092] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 527.507302] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 533.537613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.547737] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 544.619940] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 553.222821] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 561.633904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.640984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.653330] Lustre: Skipped 1 previous similar message [ 561.663062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 565.877732] Lustre: server umount lustre-MDT0000 complete [ 568.814921] LustreError: 18841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.839345] LustreError: 18841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 569.765298] LustreError: 18826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216596 with bad export cookie 7158135307827159800 [ 569.769211] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.795549] LustreError: 18826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.214070] Lustre: server umount lustre-MDT0001 complete [ 584.897798] Lustre: server umount lustre-OST0000 complete [ 614.120037] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 626.058472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.536496] LustreError: 22890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.644424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.649958] Lustre: Skipped 2 previous similar messages [ 632.041416] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 641.493541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.669883] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 649.626354] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 656.056220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 662.829959] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 672.066159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.346530] Lustre: lustre-OST0001: new disk, initializing [ 672.352898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 674.069759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 674.075809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 674.209527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 679.161616] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 688.393516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 691.629521] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 713.699255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 713.712258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.736786] Lustre: Skipped 2 previous similar messages [ 713.766363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 713.777419] Lustre: Skipped 3 previous similar messages [ 716.702315] Lustre: server umount lustre-MDT0000 complete [ 720.355336] LustreError: 22885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.381171] LustreError: 22885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 720.630187] LustreError: 22872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216746 with bad export cookie 7158135307827160647 [ 720.636292] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.645958] LustreError: 22872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.232055] Lustre: server umount lustre-MDT0001 complete [ 735.860908] Lustre: server umount lustre-OST0000 complete [ 750.975503] Lustre: server umount lustre-OST0001 complete [ 768.958909] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 780.617886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.749698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.758948] Lustre: Skipped 3 previous similar messages [ 786.914911] LustreError: 28151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.929181] LustreError: 28151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 788.381638] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 797.948144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.083147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 806.516328] Lustre: 29266:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 814.159052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 821.247327] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 830.729615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 838.675505] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 847.223575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 851.428821] Lustre: 31116:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 865.249096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 865.259894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.275139] Lustre: Skipped 3 previous similar messages [ 865.280084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.282782] Lustre: Skipped 4 previous similar messages [ 868.793352] Lustre: server umount lustre-MDT0000 complete [ 872.437619] LustreError: 28885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775216898 with bad export cookie 7158135307827161942 [ 872.444683] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.448809] LustreError: 28885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 872.867926] Lustre: server umount lustre-MDT0001 complete [ 886.945834] Lustre: server umount lustre-OST0000 complete [ 901.803247] Lustre: server umount lustre-OST0001 complete [ 911.480707] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 07:48:55 (1775216935) [ 942.752138] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 954.832745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.306857] LustreError: 33905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 955.324572] LustreError: 33905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 960.750962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 969.335054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.635684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 977.526418] Lustre: 35021:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 984.762448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.852969] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1000.920599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.516712] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1017.503705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1021.528394] Lustre: 36874:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1032.163757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.173437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.178635] Lustre: Skipped 4 previous similar messages [ 1032.190446] Lustre: Skipped 7 previous similar messages [ 1037.966795] Lustre: server umount lustre-MDT0000 complete [ 1043.020975] LustreError: 33885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217069 with bad export cookie 7158135307827163048 [ 1043.022138] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.033364] LustreError: 33885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1043.503666] Lustre: server umount lustre-MDT0001 complete [ 1058.484678] Lustre: server umount lustre-OST0000 complete [ 1072.193659] Lustre: server umount lustre-OST0001 complete [ 1082.782848] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 07:51:47 (1775217107) [ 1102.603796] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1115.063405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.886544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.892604] Lustre: Skipped 7 previous similar messages [ 1120.155931] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1130.832026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.707624] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1141.305650] Lustre: 40118:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1148.438777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.532321] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1165.044026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1172.334407] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1181.031339] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1184.954508] Lustre: 41971:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1190.968946] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1190.973707] Lustre: Skipped 1 previous similar message [ 1196.514331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.539923] Lustre: Skipped 4 previous similar messages [ 1200.226604] Lustre: server umount lustre-MDT0000 complete [ 1204.216333] LustreError: 40502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217230 with bad export cookie 7158135307827164154 [ 1204.220246] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.238265] LustreError: 40502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1210.665800] Lustre: server umount lustre-MDT0001 complete [ 1219.866450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.641981] LustreError: 40482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.682967] LustreError: 40482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 1235.318135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1240.061990] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1244.439063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.787126] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1250.290151] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1250.296076] Lustre: Skipped 1 previous similar message [ 1255.835383] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1258.090916] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1262.779708] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1264.515337] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1270.763145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1276.323597] Lustre: server umount lustre-MDT0000 complete [ 1280.024377] LustreError: 40502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217306 with bad export cookie 7158135307827165344 [ 1280.031146] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.040689] LustreError: 40502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1280.266893] Lustre: server umount lustre-MDT0001 complete [ 1294.439062] Lustre: server umount lustre-OST0000 complete [ 1297.386428] Lustre: 3657:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217307/real 1775217307] req@ffff91bac2f23480 x1861449017649408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1298.187898] Lustre: server umount lustre-OST0001 complete [ 1307.759853] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 07:55:32 (1775217332) [ 1324.192512] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1336.297931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.195164] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1350.604227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.392538] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1357.505133] Lustre: 47468:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1363.703676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.891101] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1380.417044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1387.422447] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1394.948753] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1404.369971] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1417.704916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1417.713516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.732943] Lustre: Skipped 8 previous similar messages [ 1417.746095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.750205] Lustre: Skipped 22 previous similar messages [ 1423.618574] Lustre: server umount lustre-MDT0000 complete [ 1427.479750] LustreError: 46329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775217453 with bad export cookie 7158135307827166135 [ 1427.481843] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1427.489680] LustreError: 46329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1427.802305] Lustre: server umount lustre-MDT0001 complete [ 1442.262079] Lustre: server umount lustre-OST0000 complete [ 1444.319111] Lustre: 3658:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775217454/real 1775217454] req@ffff91bbc356df80 x1861449017710336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775217470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1446.213241] Lustre: server umount lustre-OST0001 complete [ 1455.505721] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 07:58:00 (1775217480) [ 1457.652853] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1459.053585] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 07:58:04 (1775217484) [ 1469.225608] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 1478.219583] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1507.815916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.089932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1508.168571] Lustre: lustre-MDT0000: new disk, initializing [ 1508.262369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1512.797590] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1520.894982] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1528.686979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1529.143933] Lustre: lustre-OST0000: new disk, initializing [ 1529.151421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1529.157887] Lustre: Skipped 1 previous similar message [ 1530.739192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1530.751725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1530.790440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1535.772566] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1543.723868] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1545.899444] Lustre: DEBUG MARKER: create test files [ 1582.049664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.569417] Lustre: server umount lustre-OST0000 complete [ 1588.120379] Lustre: server umount lustre-MDT0000 complete [ 1600.869391] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1604.217720] Key type lgssc unregistered [ 1604.469839] LNet: 54875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.479899] LNetError: 54875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.499684] LNet: Removed LNI 192.168.201.109@tcp [ 1605.589134] Key type .llcrypt unregistered [ 1605.590974] Key type ._llcrypt unregistered [ 1636.959399] Key type ._llcrypt registered [ 1636.964458] Key type .llcrypt registered [ 1637.095520] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1638.179895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.297486] alg: No test for adler32 (adler32-zlib) [ 1639.676748] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1640.035925] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1641.713789] Key type lgssc registered [ 1643.031198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.864737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1650.876700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.479665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.758456] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1664.168444] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.505851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1672.931907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.054058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1677.153238] Lustre: 57509:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1677.164354] Lustre: 57509:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1677.173110] Lustre: 57509:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1677.194111] Lustre: 57509:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1677.208242] Lustre: 57509:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1677.227208] Lustre: 57509:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1679.932812] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1686.181496] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1691.637280] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1697.073900] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1706.298635] Lustre: server umount lustre-OST0000 complete [ 1719.982052] Lustre: server umount lustre-MDT0000 complete [ 1732.992256] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1736.041040] Key type lgssc unregistered [ 1736.364455] LNet: 58991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.374208] LNetError: 58991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.391030] LNet: Removed LNI 192.168.201.109@tcp [ 1737.313576] Key type .llcrypt unregistered [ 1737.315287] Key type ._llcrypt unregistered [ 1770.036239] Key type ._llcrypt registered [ 1770.038614] Key type .llcrypt registered [ 1770.227606] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1772.477124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1772.523916] alg: No test for adler32 (adler32-zlib) [ 1773.752956] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1774.084678] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1775.808364] Key type lgssc registered [ 1777.846058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.843793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.872809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.617295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.987183] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1802.541332] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.157833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1810.599545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.724592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1814.847392] Lustre: 61695:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 1814.857531] Lustre: 61695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/1, destroy: 0/0/0 [ 1814.863901] Lustre: 61695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1814.874974] Lustre: 61695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1814.881038] Lustre: 61695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1814.893384] Lustre: 61695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1816.688422] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1822.338403] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1828.094915] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1840.610205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.611950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.640715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.047201] Lustre: server umount lustre-OST0000 complete [ 1848.133261] Lustre: server umount lustre-MDT0000 complete [ 1860.402685] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1863.799618] Key type lgssc unregistered [ 1864.172726] LNet: 63125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.181476] LNetError: 63125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.208876] LNet: Removed LNI 192.168.201.109@tcp [ 1865.148896] Key type .llcrypt unregistered [ 1865.152617] Key type ._llcrypt unregistered [ 1883.882181] Key type ._llcrypt registered [ 1883.885264] Key type .llcrypt registered [ 1884.021344] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 1902.021780] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1903.894625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.944203] alg: No test for adler32 (adler32-zlib) [ 1905.045726] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 1905.295719] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1906.999222] Key type lgssc registered [ 1908.830392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.192190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.232567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.586811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1950.637405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1950.741119] Lustre: lustre-MDT0000: new disk, initializing [ 1950.843176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.862365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1955.096054] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1966.304850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.425784] Lustre: 66246:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1966.466158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1966.470066] Lustre: Skipped 1 previous similar message [ 1966.515425] Lustre: lustre-MDT0001: new disk, initializing [ 1966.565495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.592679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1966.607493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1970.756236] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1978.131777] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1981.640408] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1987.624632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1987.852664] Lustre: lustre-OST0000: new disk, initializing [ 1987.858244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1987.927455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.822801] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1995.851898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1995.857505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1995.954209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2001.460861] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2006.195569] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2006.443713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2011.348432] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2011.645708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2016.737781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.754680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2016.766217] Lustre: Skipped 1 previous similar message [ 2016.799430] Lustre: Skipped 1 previous similar message [ 2019.620795] Lustre: server umount lustre-OST0000 complete [ 2022.883910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2022.902953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.933191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.988561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.990922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.209412] Lustre: server umount lustre-MDT0000 complete [ 2035.017449] LustreError: 66238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218061 with bad export cookie 8095644643194678136 [ 2035.021035] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.026853] LustreError: 66238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2035.365308] Lustre: server umount lustre-MDT0001 complete [ 2048.361908] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 08:07:52 (1775218072) [ 2064.725428] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2112.987340] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2125.806302] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2172.344887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.635538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2172.690152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2172.803609] Lustre: lustre-MDT0000: new disk, initializing [ 2172.942337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.984171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2178.135952] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2191.686356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.911312] Lustre: 73834:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2191.980882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2191.984515] Lustre: Skipped 1 previous similar message [ 2192.067617] Lustre: lustre-MDT0001: new disk, initializing [ 2192.218257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2192.273058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2192.296582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2197.962329] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2206.362866] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2210.494831] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2217.767589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2218.053474] Lustre: lustre-OST0000: new disk, initializing [ 2218.056715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2218.193180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.815987] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2227.847344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2227.855345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2228.026461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2236.030020] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.925425] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2242.219661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2247.664750] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2248.004648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2253.801015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.830383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.835460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.857518] Lustre: Skipped 2 previous similar messages [ 2256.825911] Lustre: server umount lustre-OST0000 complete [ 2264.039324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.040769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.042605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.046909] Lustre: Skipped 2 previous similar messages [ 2274.274634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.292120] Lustre: Skipped 4 previous similar messages [ 2275.295164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2275.501715] Lustre: server umount lustre-MDT0000 complete [ 2279.008024] LustreError: 73826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218305 with bad export cookie 8095644643194679018 [ 2279.019418] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.020274] LustreError: 73826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.296446] Lustre: server umount lustre-MDT0001 complete [ 2289.401947] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 08:11:54 (1775218314) [ 2298.064548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.864206] LustreError: 77840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.893224] LustreError: 77840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2298.978149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.988458] LustreError: 77841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.667392] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2309.094891] LustreError: 77840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.210125] LustreError: 77841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.414122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.926647] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2324.732245] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2329.213311] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2338.164267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.626303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.640902] Lustre: Skipped 1 previous similar message [ 2344.524337] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2350.718838] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2354.673864] Lustre: 80178:0:(genops.c:1793:obd_export_evict_by_uuid()) MGS: evicting b6632077-75d5-4df7-9612-cd85cb787b36 at adminstrative request [ 2355.171840] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2355.209402] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2365.426064] Lustre: MGS: Client b6632077-75d5-4df7-9612-cd85cb787b36 (at 0@lo) reconnecting [ 2365.446254] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x705981e91169b25c to 0x705981e91169b4fc [ 2365.474308] Lustre: MGC192.168.201.109@tcp: Connection restored to 0@lo (at 0@lo) [ 2393.008735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2393.257430] Lustre: lustre-OST0001: new disk, initializing [ 2393.261961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2399.393200] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2402.404892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2402.412987] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2402.586521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2407.194723] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2413.026048] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.045223] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.053450] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2413.068953] Lustre: Skipped 1 previous similar message [ 2415.792809] Lustre: server umount lustre-OST0001 complete [ 2418.151989] LustreError: 79439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.175222] LustreError: 79439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2421.732655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.745038] LustreError: Skipped 1 previous similar message [ 2426.237826] Lustre: server umount lustre-OST0000 complete [ 2432.482149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2432.491938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.517163] Lustre: Skipped 3 previous similar messages [ 2436.489272] Lustre: server umount lustre-MDT0000 complete [ 2438.624469] LustreError: 77835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.655282] LustreError: 77835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2440.427273] LustreError: 77822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218466 with bad export cookie 8095644643194680572 [ 2440.435162] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.441172] LustreError: 77822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2440.950940] Lustre: server umount lustre-MDT0001 complete [ 2450.435581] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 08:14:34 (1775218474) [ 2452.402326] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2454.476249] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 08:14:38 (1775218478) [ 2456.132307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2458.350782] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 08:14:42 (1775218482) [ 2460.041917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2461.920124] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 08:14:46 (1775218486) [ 2463.613769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2465.726808] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 08:14:50 (1775218490) [ 2467.595543] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2474.604724] loop: module loaded [ 2474.618416] loop0: detected capacity change from 0 to 4294967296 [ 2474.675313] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2476.631640] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2480.300751] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2484.700362] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2512.869353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.911392] LustreError: 84569:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2512.916872] LustreError: 84569:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2512.926039] LustreError: 84569:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2512.940450] LustreError: 84569:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2533.997546] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2543.463757] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2582.103626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.429768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2582.460710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2582.568316] Lustre: lustre-MDT0000: new disk, initializing [ 2582.695203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.700816] Lustre: Skipped 1 previous similar message [ 2582.729166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2587.835242] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2598.985681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2599.115588] Lustre: 87402:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2599.399062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2603.899679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2611.315963] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.328613] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2621.905542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.578586] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2630.759218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2630.909708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2636.839485] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2642.051379] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2642.213398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2647.308781] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2647.597409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2651.617098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2651.619188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.635343] LustreError: Skipped 1 previous similar message [ 2651.636206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.636211] Lustre: Skipped 7 previous similar messages [ 2651.697234] Lustre: Skipped 2 previous similar messages [ 2656.325222] Lustre: server umount lustre-OST0000 complete [ 2666.888547] Lustre: server umount lustre-MDT0000 complete [ 2666.987052] LustreError: 88532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.016864] LustreError: 88532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2671.511036] LustreError: 88180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775218697 with bad export cookie 8095644643194681188 [ 2671.515311] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.526097] LustreError: 88180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2671.732123] Lustre: server umount lustre-MDT0001 complete [ 2681.230559] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 08:18:25 (1775218705) [ 2687.370563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.021852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2703.811759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.845841] LustreError: 91410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.866947] LustreError: 91410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2708.576678] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2715.006799] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.033358] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2725.961064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.313057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.317933] Lustre: Skipped 4 previous similar messages [ 2732.833258] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2739.176106] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2744.214962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2744.467279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2749.409878] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2749.687390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2756.964492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.130305] Lustre: lustre-OST0001: new disk, initializing [ 2757.133891] Lustre: Skipped 2 previous similar messages [ 2757.139816] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2757.147825] Lustre: Skipped 3 previous similar messages [ 2764.214897] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2764.379499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2764.389556] Lustre: Skipped 2 previous similar messages [ 2764.408113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2764.598689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2772.989734] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2778.234879] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2778.551936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2784.121617] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2784.524987] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2806.323326] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2832.084281] Lustre: Failing over lustre-MDT0000 [ 2832.353126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2832.368664] LustreError: Skipped 1 previous similar message [ 2832.376122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.403166] Lustre: Skipped 2 previous similar messages [ 2832.419153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.449713] Lustre: Skipped 4 previous similar messages [ 2832.675697] Lustre: server umount lustre-MDT0000 complete [ 2833.246952] LustreError: 92397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.259937] LustreError: 92397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2852.831109] Lustre: 64469:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775218862/real 1775218862] req@000000009d83e8c1 x1861450917484800/t0(0) o400->MGC192.168.201.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775218878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2852.874102] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.726601] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2857.728911] LDISKFS-fs (dm-0): recovery complete [ 2857.733821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.073479] LustreError: 64468:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000070c3a968 x1861450917493760/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2863.576599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2864.971291] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2864.973115] Lustre: 96333:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2868.718407] Lustre: 91411:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2868.724688] Lustre: 91411:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 2868.742964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2868.863264] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2869.047386] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2869.199161] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2869.553684] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2869.574988] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 1 previous similar message [ 2869.919113] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2869.937412] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 1 previous similar message [ 2869.944656] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2869.963250] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 7 previous similar messages [ 2870.617193] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2870.638597] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 2 previous similar messages [ 2870.991290] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2871.002297] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 2 previous similar messages [ 2871.984647] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2872.000039] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 5 previous similar messages [ 2872.649166] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2872.667542] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 5 previous similar messages [ 2872.999147] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2873.013972] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 5 previous similar messages [ 2876.193937] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2876.218145] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 12 previous similar messages [ 2876.871761] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2876.888128] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 12 previous similar messages [ 2877.207129] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2877.217368] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 12 previous similar messages [ 2884.208401] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2884.235324] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 24 previous similar messages [ 2884.896815] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2884.912560] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 24 previous similar messages [ 2885.239528] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2885.251303] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 24 previous similar messages [ 2900.231966] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2900.257943] Lustre: 97114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 49 previous similar messages [ 2900.906513] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2900.933950] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 49 previous similar messages [ 2901.263845] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2901.278044] LustreError: 97114:0:(ldlm_lib.c:2681:replay_request_or_update()) Skipped 49 previous similar messages [ 2924.247146] Lustre: 97114:0:(ldlm_lib.c:2377:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2924.256908] Lustre: 97114:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2924.273616] Lustre: 97114:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f4faa42b-3aac-4ae5-a1c3-5e141000f98d@192.168.201.9@tcp [ 2924.293338] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2924.296636] Lustre: 97114:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@00000000549393fe x1861450898410880/t0(8589934769) o101->f4faa42b-3aac-4ae5-a1c3-5e141000f98d@192.168.201.9@tcp:296/0 lens 592/0 e 0 to 0 dl 1775218961 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2924.358516] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2924.362615] Lustre: lustre-MDT0000: Denying connection for new client f4faa42b-3aac-4ae5-a1c3-5e141000f98d (at 192.168.201.9@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2924.395448] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2924.446749] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2924.449637] Lustre: Skipped 2 previous similar messages [ 2924.462437] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2924.533243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2924.535373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2939.361806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2944.415583] Lustre: server umount lustre-OST0000 complete [ 2960.865543] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2960.892127] Lustre: Skipped 9 previous similar messages [ 2962.399357] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.709871] Lustre: server umount lustre-OST0001 complete [ 2970.591875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.606701] Lustre: Skipped 7 previous similar messages [ 2972.599773] Lustre: server umount lustre-MDT0000 complete [ 2976.050414] LustreError: 91390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219002 with bad export cookie 8095644643194732407 [ 2976.056783] LustreError: 91390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2976.057193] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.412591] Lustre: server umount lustre-MDT0001 complete [ 2984.538564] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 08:23:29 (1775219009) [ 2992.066546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2992.078720] Lustre: *** cfs_fail_loc=197, val=0*** [ 3007.904968] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3013.028087] LustreError: 99081:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 3018.257451] Lustre: *** cfs_fail_loc=197, val=0*** [ 3018.264616] Lustre: Skipped 65 previous similar messages [ 3038.815397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.823929] Lustre: Skipped 2 previous similar messages [ 3045.221640] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3051.123721] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3053.212869] Lustre: server umount lustre-OST0000 complete [ 3061.852741] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 08:24:46 (1775219086) [ 3082.635890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.238951] LustreError: 100590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.271393] LustreError: 100590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 48 previous similar messages [ 3088.061090] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3098.284441] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3103.103922] LustreError: 100621:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3103.121130] LustreError: 100621:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3103.135545] LustreError: 100621:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 3103.814742] Lustre: server umount lustre-MDT0000 complete [ 3135.479082] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3146.080196] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3184.876850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.118156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3185.146653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3185.268930] Lustre: lustre-MDT0000: new disk, initializing [ 3185.425679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3189.828554] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3201.277904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.414890] Lustre: 104645:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3201.699057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3205.850812] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3213.978679] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3218.114671] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3226.222766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3232.506176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3234.491897] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3243.097849] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3248.240399] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3248.471312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3253.192785] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3253.468874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3258.338139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.343601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3258.357465] Lustre: Skipped 2 previous similar messages [ 3258.364814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3258.369499] Lustre: Skipped 4 previous similar messages [ 3258.388605] LustreError: Skipped 4 previous similar messages [ 3261.535668] Lustre: server umount lustre-OST0000 complete [ 3271.599551] Lustre: server umount lustre-MDT0000 complete [ 3275.789899] LustreError: 104639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219302 with bad export cookie 8095644643194742977 [ 3275.791974] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.807011] LustreError: 104639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3276.068322] Lustre: server umount lustre-MDT0001 complete [ 3285.136741] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 08:28:29 (1775219309) [ 3294.625919] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3297.950721] Key type lgssc unregistered [ 3298.183280] LNet: 108718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3298.192853] LNetError: 108718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3298.205692] LNet: Removed LNI 192.168.201.109@tcp [ 3299.045354] Key type .llcrypt unregistered [ 3299.053121] Key type ._llcrypt unregistered [ 3317.835676] Key type ._llcrypt registered [ 3317.837935] Key type .llcrypt registered [ 3317.972251] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3333.833986] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3336.086203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3336.167877] alg: No test for adler32 (adler32-zlib) [ 3337.316268] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3337.746816] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3339.498871] Key type lgssc registered [ 3341.478829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3386.294554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3386.329840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.768101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3387.803284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3387.951626] Lustre: lustre-MDT0000: new disk, initializing [ 3388.090716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.122506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3392.666963] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3404.032735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3404.213783] Lustre: lustre-OST0000: new disk, initializing [ 3404.216358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3404.220307] Lustre: Skipped 1 previous similar message [ 3404.258809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.894184] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3414.161065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3414.179139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3414.223352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3420.324563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.497784] Lustre: lustre-OST0001: new disk, initializing [ 3420.501887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3420.595150] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.004873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3427.014384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3427.041695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3427.683060] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3437.538138] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.552039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3441.593685] Lustre: server umount lustre-MDT0000 complete [ 3447.935989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3453.049881] LustreError: 112853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219479 with bad export cookie 13702120566356434788 [ 3453.059168] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3453.162853] Lustre: server umount lustre-OST0000 complete [ 3457.898322] Lustre: server umount lustre-OST0001 complete [ 3468.812775] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 08:31:33 (1775219493) [ 3480.481926] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3506.471111] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 08:32:11 (1775219531) [ 3548.473457] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3561.388613] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3600.488608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.831638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3600.873763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3600.966466] Lustre: lustre-MDT0000: new disk, initializing [ 3601.074556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.091331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3606.834292] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3618.840876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.954707] Lustre: 120378:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3619.008502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3619.013501] Lustre: Skipped 1 previous similar message [ 3619.117411] Lustre: lustre-MDT0001: new disk, initializing [ 3619.218534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3619.255982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3619.270470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3624.310475] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3632.626421] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3636.460794] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3644.055981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.534351] Lustre: lustre-OST0000: new disk, initializing [ 3644.544564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3644.630739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.030548] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3652.198960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3652.210890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3652.409640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3659.609983] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3664.265885] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3664.488173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3668.915240] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3669.033481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3673.066444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.072110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.088704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.093188] Lustre: Skipped 2 previous similar messages [ 3673.107320] Lustre: Skipped 2 previous similar messages [ 3677.867616] Lustre: server umount lustre-OST0000 complete [ 3683.300697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.305203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.333410] Lustre: Skipped 1 previous similar message [ 3683.361290] Lustre: Skipped 1 previous similar message [ 3687.982812] Lustre: server umount lustre-MDT0000 complete [ 3692.677489] LustreError: 122128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219718 with bad export cookie 13702120566356436013 [ 3692.694341] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3692.698394] LustreError: 122128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3693.162701] Lustre: server umount lustre-MDT0001 complete [ 3701.813847] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 08:35:26 (1775219726) [ 3708.050824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.420635] LustreError: 124386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.443885] LustreError: 124386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3708.500982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.518626] LustreError: 124387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.583422] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3718.640368] LustreError: 124386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3722.782416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.041882] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3732.861892] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3736.807843] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3743.830736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3744.318190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.333808] Lustre: Skipped 1 previous similar message [ 3750.016683] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3757.870795] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3764.056911] Lustre: *** cfs_fail_loc=159, val=0*** [ 3764.060844] Lustre: Skipped 1 previous similar message [ 3780.468737] Lustre: lustre-MDT0000: Client 30ec85e7-6280-48ec-9aa8-479245ec2ffd (at 192.168.201.9@tcp) reconnecting [ 3782.045895] Lustre: *** cfs_fail_loc=159, val=0*** [ 3782.055724] Lustre: Skipped 4 previous similar messages [ 3798.441591] Lustre: lustre-MDT0000: Client 30ec85e7-6280-48ec-9aa8-479245ec2ffd (at 192.168.201.9@tcp) reconnecting [ 3820.005813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.010966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.035106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.069422] LustreError: Skipped 2 previous similar messages [ 3821.435248] Lustre: server umount lustre-OST0000 complete [ 3825.633168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3825.637811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.648772] Lustre: Skipped 1 previous similar message [ 3825.653777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.659232] Lustre: Skipped 1 previous similar message [ 3832.207873] Lustre: server umount lustre-MDT0000 complete [ 3835.360161] LustreError: 125357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3835.374547] LustreError: 125357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3835.648853] LustreError: 124366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775219861 with bad export cookie 13702120566356436888 [ 3835.649324] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3835.655240] LustreError: 124366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3835.870731] Lustre: server umount lustre-MDT0001 complete [ 3849.484566] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3852.823499] Key type lgssc unregistered [ 3853.252504] LNet: 127812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3853.265949] LNetError: 127812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3854.311888] LNet: Removed LNI 192.168.201.109@tcp [ 3855.772714] Key type .llcrypt unregistered [ 3855.788549] Key type ._llcrypt unregistered [ 3883.823739] Key type ._llcrypt registered [ 3883.826239] Key type .llcrypt registered [ 3883.939399] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3884.966069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.979606] alg: No test for adler32 (adler32-zlib) [ 3886.269590] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 3886.672140] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3888.409919] Key type lgssc registered [ 3889.686906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.340425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3898.358476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.148145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.882543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3910.629084] LustreError: 129219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.654929] LustreError: 129219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3915.745975] LustreError: 129220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3916.369986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.862792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3922.272350] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3927.601273] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3931.745270] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3939.245260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.493510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.668119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3946.492371] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3953.025320] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3962.711862] Lustre: *** cfs_fail_loc=159, val=0*** [ 3978.106071] Lustre: lustre-MDT0000: Client 2da5e7b2-b5f1-4b19-b7a4-fafa0b0fcf6b (at 192.168.201.9@tcp) reconnecting [ 3980.301832] Lustre: *** cfs_fail_loc=159, val=0*** [ 3996.550882] Lustre: lustre-MDT0001: Client 2da5e7b2-b5f1-4b19-b7a4-fafa0b0fcf6b (at 192.168.201.9@tcp) reconnecting [ 3997.772577] Lustre: *** cfs_fail_loc=159, val=0*** [ 3997.779566] Lustre: Skipped 3 previous similar messages [ 4013.482254] Lustre: lustre-MDT0001: Client 2da5e7b2-b5f1-4b19-b7a4-fafa0b0fcf6b (at 192.168.201.9@tcp) reconnecting [ 4019.274820] Lustre: 131916:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4022.098981] Lustre: *** cfs_fail_loc=159, val=0*** [ 4022.104499] Lustre: Skipped 4 previous similar messages [ 4038.580711] Lustre: lustre-MDT0000: Client f8e30ca4-35dd-4dec-a1fc-7b628bde0d59 (at 192.168.201.9@tcp) reconnecting [ 4040.663357] Lustre: *** cfs_fail_loc=159, val=0*** [ 4040.668992] Lustre: Skipped 13 previous similar messages [ 4055.936921] Lustre: lustre-MDT0000: Client f8e30ca4-35dd-4dec-a1fc-7b628bde0d59 (at 192.168.201.9@tcp) reconnecting [ 4057.536688] Lustre: 132215:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4079.072883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.074913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.088692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.114543] Lustre: Skipped 1 previous similar message [ 4084.196163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.207849] Lustre: Skipped 2 previous similar messages [ 4089.314144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.329645] Lustre: Skipped 1 previous similar message [ 4090.847223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4091.085832] Lustre: server umount lustre-OST0000 complete [ 4095.969891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4095.976675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.994129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.558677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4104.672889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.686723] Lustre: Skipped 3 previous similar messages [ 4109.281754] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.482388] Lustre: server umount lustre-MDT0000 complete [ 4109.799752] LustreError: 129930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4109.820498] LustreError: 129930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4112.919808] LustreError: 129201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220139 with bad export cookie 12379399090580801088 [ 4112.934688] LustreError: 129201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4112.940436] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4113.297827] Lustre: server umount lustre-MDT0001 complete [ 4125.607841] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4128.485195] Key type lgssc unregistered [ 4128.741596] LNet: 133287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4128.760150] LNetError: 133287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4129.845866] LNet: Removed LNI 192.168.201.109@tcp [ 4130.614489] Key type .llcrypt unregistered [ 4130.625184] Key type ._llcrypt unregistered [ 4155.716782] Key type ._llcrypt registered [ 4155.720749] Key type .llcrypt registered [ 4155.841600] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4157.028692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4157.173233] alg: No test for adler32 (adler32-zlib) [ 4158.279502] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4158.580888] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4160.327230] Key type lgssc registered [ 4161.640673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4170.251911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4170.264681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.868636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.827482] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4181.987367] LustreError: 134687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.009901] LustreError: 134687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4185.444110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.033038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.060886] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4197.866946] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4202.012750] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4208.451165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.025376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.137132] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4215.221318] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4217.266877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4220.178459] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4225.625269] Lustre: 136932:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4228.957698] Lustre: 136981:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4230.268711] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4231.126473] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4234.712523] Lustre: 137127:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4235.293727] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4236.505692] Lustre: 137226:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4241.894668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4241.904125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.920491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4242.912602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4242.915304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.469255] Lustre: server umount lustre-OST0000 complete [ 4252.654931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4252.663425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.677585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4252.680055] Lustre: Skipped 1 previous similar message [ 4256.656648] Lustre: server umount lustre-MDT0000 complete [ 4258.273139] LustreError: 134683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.294193] LustreError: 134683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4260.228826] LustreError: 136302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220286 with bad export cookie 13118235508297751076 [ 4260.236519] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.252396] LustreError: 136302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4260.618519] Lustre: server umount lustre-MDT0001 complete [ 4272.301730] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4275.688835] Key type lgssc unregistered [ 4276.031766] LNet: 138268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4276.035815] LNetError: 138268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4276.056599] LNet: Removed LNI 192.168.201.109@tcp [ 4277.192559] Key type .llcrypt unregistered [ 4277.199153] Key type ._llcrypt unregistered [ 4305.378360] Key type ._llcrypt registered [ 4305.383478] Key type .llcrypt registered [ 4305.555917] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4307.058682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4307.171347] alg: No test for adler32 (adler32-zlib) [ 4308.330504] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4308.595902] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4310.359239] Key type lgssc registered [ 4311.800070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4319.042122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4319.063427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.557275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.825970] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4329.962364] LustreError: 139667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.989570] LustreError: 139667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4334.905462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.073098] LustreError: 139666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.358912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4340.277958] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4345.874892] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4349.115565] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.124657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.559983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.728036] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4361.709188] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4364.821973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4367.878614] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4373.490554] Lustre: *** cfs_fail_loc=159, val=0*** [ 4389.293462] Lustre: lustre-MDT0000: Client 02587374-98d0-46c4-b6f3-1f5536624cc6 (at 192.168.201.9@tcp) reconnecting [ 4394.982987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4394.991079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.998716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4396.840199] Lustre: server umount lustre-OST0000 complete [ 4400.613268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.630107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.630424] Lustre: Skipped 2 previous similar messages [ 4400.658159] Lustre: Skipped 3 previous similar messages [ 4405.730738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.740403] Lustre: Skipped 1 previous similar message [ 4406.572948] Lustre: server umount lustre-MDT0000 complete [ 4410.190773] LustreError: 139647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220436 with bad export cookie 14800394051265079276 [ 4410.192913] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4410.200530] LustreError: 139647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.547823] Lustre: server umount lustre-MDT0001 complete [ 4420.427177] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4423.080853] Key type lgssc unregistered [ 4423.394965] LNet: 142978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4423.408976] LNetError: 142978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4423.431699] LNet: Removed LNI 192.168.201.109@tcp [ 4424.354789] Key type .llcrypt unregistered [ 4424.356773] Key type ._llcrypt unregistered [ 4447.302711] Key type ._llcrypt registered [ 4447.304123] Key type .llcrypt registered [ 4447.399068] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4448.340302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4448.352089] alg: No test for adler32 (adler32-zlib) [ 4449.332325] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4449.500939] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4451.159181] Key type lgssc registered [ 4452.147879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4459.497855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4459.515865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.224660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.798441] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4470.754412] LustreError: 144376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.769847] LustreError: 144376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4473.674725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4474.101513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.658819] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4483.949300] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4488.051188] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4494.618652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4494.944640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.059810] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4500.508308] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4505.382652] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4506.263345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4524.077020] Lustre: lustre-OST0000: evicting 97acc613-f156-4fda-81c6-26498a4903c8 (at 192.168.201.9@tcp) by administrative request [ 4524.085274] Lustre: Skipped 1 previous similar message [ 4552.678479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.679691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.693165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.723255] Lustre: Skipped 1 previous similar message [ 4557.796675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4557.801338] Lustre: Skipped 2 previous similar messages [ 4558.131318] Lustre: server umount lustre-OST0000 complete [ 4562.912929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.921843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4562.929396] Lustre: Skipped 1 previous similar message [ 4568.036158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.038556] Lustre: Skipped 2 previous similar messages [ 4573.154533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4573.166951] Lustre: Skipped 1 previous similar message [ 4576.229471] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4576.419454] Lustre: server umount lustre-MDT0000 complete [ 4578.274772] LustreError: 144377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.292397] LustreError: 144377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4579.543845] LustreError: 144357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220605 with bad export cookie 4177740613528631053 [ 4579.551514] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4579.795663] Lustre: server umount lustre-MDT0001 complete [ 4589.931284] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4592.413804] Key type lgssc unregistered [ 4592.778514] LNet: 148091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4592.785295] LNetError: 148091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4592.798715] LNet: Removed LNI 192.168.201.109@tcp [ 4593.462132] Key type .llcrypt unregistered [ 4593.463981] Key type ._llcrypt unregistered [ 4655.857385] Key type ._llcrypt registered [ 4655.860364] Key type .llcrypt registered [ 4655.961587] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4656.917316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4656.986926] alg: No test for adler32 (adler32-zlib) [ 4658.063894] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4658.226447] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4659.919276] Key type lgssc registered [ 4661.330920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4668.196217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4668.229616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.604385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.486921] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4679.147325] LustreError: 150560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4679.170287] LustreError: 150560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4681.897870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.356551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4686.263406] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4691.319522] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4694.263959] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4699.874800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4700.296765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.468366] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4707.452367] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4710.596192] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4714.619983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4723.171784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4723.179804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.197507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4725.219392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.230173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.230838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.106093] Lustre: server umount lustre-OST0000 complete [ 4733.421787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4733.430413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.451211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.453864] Lustre: Skipped 1 previous similar message [ 4735.464062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.472578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.614600] Lustre: server umount lustre-MDT0000 complete [ 4740.004378] LustreError: 150541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220766 with bad export cookie 1277761647075049006 [ 4740.009161] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.012364] LustreError: 150541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4740.253029] Lustre: server umount lustre-MDT0001 complete [ 4748.408558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4750.364221] Key type lgssc unregistered [ 4750.569778] LNet: 153748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4750.580655] LNetError: 153748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4750.590041] LNet: Removed LNI 192.168.201.109@tcp [ 4751.202942] Key type .llcrypt unregistered [ 4751.206436] Key type ._llcrypt unregistered [ 4768.158951] Key type ._llcrypt registered [ 4768.160967] Key type .llcrypt registered [ 4768.245295] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4768.893426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4768.962134] alg: No test for adler32 (adler32-zlib) [ 4770.058431] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4770.208138] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4771.903146] Key type lgssc registered [ 4772.818668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4784.484257] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 08:53:29 (1775220809) [ 4797.059171] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4806.207731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4837.903520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4837.942760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.198766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4839.233100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4839.315896] Lustre: lustre-MDT0000: new disk, initializing [ 4839.374523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.391686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4843.306165] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4851.596154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.663275] Lustre: 157939:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4851.714092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4851.723932] Lustre: Skipped 1 previous similar message [ 4851.801735] Lustre: lustre-MDT0001: new disk, initializing [ 4851.894823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4851.938042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4851.943027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4855.095228] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4861.199191] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4863.886153] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4868.841652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.077339] Lustre: lustre-OST0000: new disk, initializing [ 4869.081357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4869.161411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.457682] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4878.399044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4878.411396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4878.475143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4879.059530] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4889.058268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4889.062980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.086029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.091461] Lustre: Skipped 1 previous similar message [ 4892.251560] Lustre: server umount lustre-OST0000 complete [ 4898.272027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4898.276424] LustreError: Skipped 1 previous similar message [ 4898.279558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.286494] Lustre: Skipped 1 previous similar message [ 4898.290343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4899.299725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.301170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4899.310947] Lustre: Skipped 1 previous similar message [ 4901.483326] Lustre: server umount lustre-MDT0000 complete [ 4904.043125] LustreError: 157931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775220930 with bad export cookie 16294612309502935984 [ 4904.044255] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.053152] LustreError: 157931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4904.316903] Lustre: server umount lustre-MDT0001 complete [ 4912.712895] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4915.106977] Key type lgssc unregistered [ 4915.352854] LNet: 161451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4915.376902] LNetError: 161451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4915.389522] LNet: Removed LNI 192.168.201.109@tcp [ 4916.061608] Key type .llcrypt unregistered [ 4916.064346] Key type ._llcrypt unregistered [ 4936.154729] Key type ._llcrypt registered [ 4936.156506] Key type .llcrypt registered [ 4936.267622] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4937.067961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.136661] alg: No test for adler32 (adler32-zlib) [ 4938.136607] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 4938.309897] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4939.935199] Key type lgssc registered [ 4940.794684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4946.444481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4946.461214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.785485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.595313] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4956.676373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.946213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.376277] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4962.752878] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4965.551121] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4970.622560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4970.819920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.848335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4975.014858] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4978.374669] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4987.449157] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4987.458302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.467195] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4987.473289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.481641] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4988.056304] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4988.069750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.085157] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4988.095885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.110825] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4988.690381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.205451] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4989.214746] Lustre: Skipped 1 previous similar message [ 4989.224371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.237574] Lustre: Skipped 1 previous similar message [ 4989.242134] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4989.250745] Lustre: Skipped 1 previous similar message [ 4989.255435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.264936] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4989.274549] Lustre: Skipped 1 previous similar message [ 4989.856666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.541374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.175336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.711618] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4991.718891] Lustre: Skipped 3 previous similar messages [ 4991.727662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.733463] Lustre: Skipped 3 previous similar messages [ 4991.736221] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4991.739444] Lustre: Skipped 3 previous similar messages [ 4991.745300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.758272] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4991.764890] Lustre: Skipped 3 previous similar messages [ 4992.476719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.032394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.691842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.291626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.956286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.494961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.097979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4996.101734] Lustre: Skipped 6 previous similar messages [ 4996.117276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.123540] Lustre: Skipped 6 previous similar messages [ 4996.126545] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4996.133620] Lustre: Skipped 6 previous similar messages [ 4996.139666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.147393] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4996.157584] Lustre: Skipped 6 previous similar messages [ 4996.747820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.430315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.211386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.918876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.581641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.394884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.101353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.852154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.623357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.345677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.025415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.743608] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5004.751767] Lustre: Skipped 11 previous similar messages [ 5004.767542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.778327] Lustre: Skipped 11 previous similar messages [ 5004.780628] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5004.782886] Lustre: Skipped 11 previous similar messages [ 5004.785418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.790815] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5004.792515] Lustre: Skipped 11 previous similar messages [ 5005.406439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.112176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.818303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.587298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.283774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.997101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.680108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.420367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.241165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.947492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.695602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.524044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.264050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.994573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.746372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.477688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.202816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.807566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.417812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.177517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.767694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.490456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.105824] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5021.110549] Lustre: Skipped 22 previous similar messages [ 5021.127772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.134743] Lustre: Skipped 22 previous similar messages [ 5021.137866] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5021.142546] Lustre: Skipped 22 previous similar messages [ 5021.145714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.150941] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5021.155240] Lustre: Skipped 22 previous similar messages [ 5021.849185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.387411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.027860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.683300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.324848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.961290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.758704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.415266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.162210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.735779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.512847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.228448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.864474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.492326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.268539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.926315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.683871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.357942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.043240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.785863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.647235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.503905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.291343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.976835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.754981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.494514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.240484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.914323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.703784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.481247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.358678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.224801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.867256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.475924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.138482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.844329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.540254] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5047.561726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.581130] LustreError: 162860:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5048.243527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.861745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.461157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.174240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.179533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5064.179820] Lustre: Skipped 40 previous similar messages [ 5064.180517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5067.388727] Lustre: server umount lustre-OST0000 complete [ 5074.406697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5074.412379] Lustre: Skipped 2 previous similar messages [ 5076.431125] Lustre: server umount lustre-MDT0000 complete [ 5078.882304] LustreError: 162822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221105 with bad export cookie 11477320980333465694 [ 5078.891655] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.897342] LustreError: 162822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5079.110705] Lustre: server umount lustre-MDT0001 complete [ 5087.345473] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5089.313191] Key type lgssc unregistered [ 5089.538072] LNet: 170966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5089.547353] LNetError: 170966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5089.564762] LNet: Removed LNI 192.168.201.109@tcp [ 5090.102415] Key type .llcrypt unregistered [ 5090.104805] Key type ._llcrypt unregistered [ 5107.410779] Key type ._llcrypt registered [ 5107.413825] Key type .llcrypt registered [ 5107.490379] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5108.162448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5108.196479] alg: No test for adler32 (adler32-zlib) [ 5109.154708] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5109.334598] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5111.039168] Key type lgssc registered [ 5111.915473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5117.585721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5117.601655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5119.013091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.334975] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5128.684335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.961192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5132.219508] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5136.298389] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5138.976650] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5144.737597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5145.071986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.094363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1268 to 0x280000401:1313) [ 5150.249188] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5154.463575] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5159.671602] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5161.057122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.072468] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5161.076149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.082306] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5161.086207] LustreError: 172363:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 5164.521959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.528254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5164.541882] Lustre: Skipped 1 previous similar message [ 5164.542445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5164.569345] Lustre: Skipped 1 previous similar message [ 5167.447146] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 5168.766029] Lustre: server umount lustre-OST0000 complete [ 5181.135124] LustreError: 172363:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 5201.168737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5201.461576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5204.526905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.540850] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.540878] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5204.570793] Lustre: Skipped 1 previous similar message [ 5206.082823] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5211.160283] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5212.474235] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5215.527819] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff998420ca1000.ost_server_uuid 50 [ 5216.818912] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff998420ca1000.ost_server_uuid in FULL state after 0 sec [ 5222.371273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5222.387259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.400914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5224.929172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5224.933045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.935699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5224.949708] Lustre: Skipped 1 previous similar message [ 5226.512640] Lustre: server umount lustre-OST0000 complete [ 5230.054132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.074731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5235.714948] Lustre: server umount lustre-MDT0000 complete [ 5238.157201] LustreError: 172324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221264 with bad export cookie 18195279492134260757 [ 5238.159939] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5238.165193] LustreError: 172324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5238.369955] Lustre: server umount lustre-MDT0001 complete [ 5244.960406] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5246.819448] Key type lgssc unregistered [ 5247.055283] LNet: 176999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5247.059760] LNetError: 176999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5247.070656] LNet: Removed LNI 192.168.201.109@tcp [ 5247.716247] Key type .llcrypt unregistered [ 5247.718029] Key type ._llcrypt unregistered [ 5268.235271] Key type ._llcrypt registered [ 5268.236605] Key type .llcrypt registered [ 5268.310445] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5268.971449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5268.989968] alg: No test for adler32 (adler32-zlib) [ 5269.962932] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5270.138545] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5271.791778] Key type lgssc registered [ 5272.658400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5279.067525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5279.082247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.452147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.585747] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5289.331227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.510632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5291.934314] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5293.873895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5298.152348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5298.391650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.521485] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5305.593624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 5307.539131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5307.650608] Lustre: lustre-OST0001: new disk, initializing [ 5307.655645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5307.712329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5311.616375] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5313.603177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5313.608927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5313.687419] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5320.653839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5326.864885] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5343.176685] Lustre: server umount lustre-MDT0000 complete [ 5344.736750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.737360] LustreError: 178683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.750880] Lustre: Skipped 3 previous similar messages [ 5344.756155] LustreError: 178683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5345.666491] LustreError: 178670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221371 with bad export cookie 620286423534775196 [ 5345.669572] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5345.677958] LustreError: 178670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5347.985608] Lustre: server umount lustre-MDT0001 complete [ 5359.996812] Lustre: server umount lustre-OST0000 complete [ 5372.270210] Lustre: server umount lustre-OST0001 complete [ 5387.608489] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5394.916037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.214080] LustreError: 184017:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.235674] LustreError: 184017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5395.281170] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.313025] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5400.545047] LustreError: 184018:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.672363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.888175] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5406.688331] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5408.180767] Lustre: 185124:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5412.453040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.720048] LustreError: 185478:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.412804] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5420.852896] LustreError: 185478:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5420.865605] LustreError: 185478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5420.871381] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 5422.152739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5422.378978] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5422.386336] Lustre: Skipped 1 previous similar message [ 5423.646774] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5427.007794] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5432.669496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5435.362223] Lustre: 186966:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5447.876887] Lustre: server umount mylustre-MDT0000 complete [ 5448.169226] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.175032] LustreError: 184013:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.180090] Lustre: Skipped 2 previous similar messages [ 5448.187812] LustreError: 184013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5450.342410] LustreError: 183993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221476 with bad export cookie 620286423534778003 [ 5450.344061] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5450.351350] LustreError: 183993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5452.611726] Lustre: server umount mylustre-MDT0001 complete [ 5465.473030] Lustre: server umount mylustre-OST0000 complete [ 5478.283023] Lustre: server umount mylustre-OST0001 complete [ 5491.175990] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5497.384034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.662517] LustreError: 189258:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.671336] LustreError: 189258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5497.723104] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.002431] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5504.911091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.245749] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5508.900217] Lustre: 190363:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5512.808689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5516.718590] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5522.310180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5522.413912] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1409) [ 5524.034779] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5525.967851] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5531.018419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5533.426307] Lustre: 192209:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5542.885989] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.893294] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5542.893783] Lustre: Skipped 4 previous similar messages [ 5542.903177] Lustre: Skipped 3 previous similar messages [ 5548.000264] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5548.279939] Lustre: server umount tfs-MDT0000 complete [ 5550.534467] LustreError: 189234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221576 with bad export cookie 620286423534779886 [ 5550.536507] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5550.545866] LustreError: 189234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5550.755357] Lustre: server umount tfs-MDT0001 complete [ 5563.361825] Lustre: server umount tfs-OST0000 complete [ 5574.991645] Lustre: server umount tfs-OST0001 complete [ 5585.726418] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5591.413273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.677279] LustreError: 194437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.688097] LustreError: 194437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5591.733302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.737880] Lustre: Skipped 3 previous similar messages [ 5594.048595] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5598.618929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.920163] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5602.503142] Lustre: 195545:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5605.917492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5609.437731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5614.370907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5616.558883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1441) [ 5616.564889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5618.209135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5623.084606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5625.307557] Lustre: 197388:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5634.107147] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 09:07:39 (1775221659) [ 5647.332352] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.334452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5647.337255] Lustre: Skipped 2 previous similar messages [ 5647.340674] Lustre: Skipped 4 previous similar messages [ 5652.451117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.457311] Lustre: Skipped 5 previous similar messages [ 5653.297652] Lustre: server umount lustre-MDT0000 complete [ 5655.532927] LustreError: 195548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221681 with bad export cookie 620286423534781552 [ 5655.533804] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5655.540289] LustreError: 195548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5657.576097] LustreError: 197760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.576384] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.583170] LustreError: 197760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5657.594670] Lustre: Skipped 1 previous similar message [ 5657.599344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5657.601695] Lustre: Skipped 1 previous similar message [ 5661.810228] Lustre: server umount lustre-MDT0001 complete [ 5678.047395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5678.102625] Lustre: server umount lustre-OST0000 complete [ 5686.527423] Lustre: server umount lustre-OST0001 complete [ 5690.183980] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5695.050483] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5713.156190] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 5713.156190] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5713.156190] [ 5713.168462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5713.290733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5713.312946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5713.355037] Lustre: lustre-MDT0000: new disk, initializing [ 5713.402449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5715.254035] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5720.789151] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 5720.789151] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5720.789151] [ 5720.811230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5720.857393] Lustre: 200537:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5720.892470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5720.894842] Lustre: Skipped 1 previous similar message [ 5720.936381] Lustre: lustre-MDT0001: new disk, initializing [ 5720.980957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5720.983495] Lustre: Skipped 4 previous similar messages [ 5720.998666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5721.003599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5722.678817] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5729.295101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.446651] Lustre: lustre-OST0000: new disk, initializing [ 5729.449172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5730.723837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5730.734001] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5730.829995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5732.998281] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5739.606805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5739.684097] Lustre: lustre-OST0001: new disk, initializing [ 5739.686069] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5741.212292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5741.223178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5741.306674] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5742.773615] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5748.870712] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 09:09:34 (1775221774) [ 5754.976976] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 09:09:40 (1775221780) [ 5767.141525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.147359] Lustre: Skipped 1 previous similar message [ 5767.150083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5767.152838] Lustre: Skipped 3 previous similar messages [ 5773.062527] Lustre: server umount lustre-OST0000 complete [ 5777.379297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5781.101148] Lustre: server umount lustre-MDT0000 complete [ 5782.793413] LustreError: 200528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221809 with bad export cookie 620286423534783113 [ 5782.797114] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.797518] LustreError: 200528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5787.617500] LustreError: 200545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.618805] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.627132] LustreError: 200545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5787.641787] Lustre: Skipped 4 previous similar messages [ 5787.645198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5787.648647] Lustre: Skipped 5 previous similar messages [ 5797.343435] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5797.457521] Lustre: server umount lustre-MDT0001 complete [ 5802.179923] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5813.441284] Lustre: server umount lustre-OST0001 complete [ 5816.704645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5825.732964] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5845.383352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.556843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5845.578895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5845.619285] Lustre: lustre-MDT0000: new disk, initializing [ 5845.672930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5847.726572] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5853.602048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.657365] Lustre: 207401:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5853.819082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5855.909032] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5859.936441] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5861.743504] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5865.640329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5865.822700] Lustre: lustre-OST0000: new disk, initializing [ 5865.826101] Lustre: Skipped 1 previous similar message [ 5865.831828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5865.834540] Lustre: Skipped 2 previous similar messages [ 5867.299543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5867.304158] Lustre: Skipped 1 previous similar message [ 5867.307466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5867.349869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5869.177632] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5873.677917] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5882.850546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5882.851925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.860938] LustreError: Skipped 1 previous similar message [ 5882.869212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5882.873747] Lustre: Skipped 3 previous similar messages [ 5884.195821] Lustre: server umount lustre-OST0000 complete [ 5892.582213] Lustre: server umount lustre-MDT0000 complete [ 5894.207197] LustreError: 207391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221920 with bad export cookie 620286423534784583 [ 5894.209475] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5894.212884] LustreError: 207391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5894.343963] Lustre: server umount lustre-MDT0001 complete [ 5898.702750] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5900.141531] Key type lgssc unregistered [ 5900.292846] LNet: 210767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5900.297408] LNetError: 210767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5900.307768] LNet: Removed LNI 192.168.201.109@tcp [ 5900.738126] Key type .llcrypt unregistered [ 5900.739777] Key type ._llcrypt unregistered [ 5913.483916] Key type ._llcrypt registered [ 5913.485276] Key type .llcrypt registered [ 5913.536418] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5913.990168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5914.040289] alg: No test for adler32 (adler32-zlib) [ 5914.943978] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5915.090265] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5916.695154] Key type lgssc registered [ 5917.119202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5920.290360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5920.299523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.534644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.272275] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5927.160066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.372594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5929.165322] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5931.630928] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5933.186395] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5936.145434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.319180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5938.902468] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5941.254304] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5941.668217] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 5942.753127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.758343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.825621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.827742] Lustre: Skipped 1 previous similar message [ 5948.584511] Lustre: server umount lustre-OST0000 complete [ 5950.945872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.947957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.952549] Lustre: Skipped 1 previous similar message [ 5956.064434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5956.072096] Lustre: Skipped 2 previous similar messages [ 5956.533273] Lustre: server umount lustre-MDT0000 complete [ 5958.531377] LustreError: 213802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775221984 with bad export cookie 12830887358911475929 [ 5958.532972] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.544498] LustreError: 213802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5958.746487] Lustre: server umount lustre-MDT0001 complete [ 5963.412338] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5964.703239] Key type lgssc unregistered [ 5964.881680] LNet: 215418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5964.884329] LNetError: 215418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5964.895848] LNet: Removed LNI 192.168.201.109@tcp [ 5965.280868] Key type .llcrypt unregistered [ 5965.282268] Key type ._llcrypt unregistered [ 5973.402775] Key type ._llcrypt registered [ 5973.403939] Key type .llcrypt registered [ 5973.464960] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5974.103100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5974.113784] alg: No test for adler32 (adler32-zlib) [ 5975.025890] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 5975.151840] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5976.791151] Key type lgssc registered [ 5977.423587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5982.269165] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5982.284921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5997.791482] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6002.911277] LustreError: 216572:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 6029.536084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6036.142175] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6040.759780] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6043.109634] Lustre: server umount lustre-OST0000 complete [ 6058.096474] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6060.472100] Key type lgssc unregistered [ 6060.674207] LNet: 218073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6060.687949] LNetError: 218073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6060.706501] LNet: Removed LNI 192.168.201.109@tcp [ 6061.220711] Key type .llcrypt unregistered [ 6061.224897] Key type ._llcrypt unregistered [ 6086.294685] Key type ._llcrypt registered [ 6086.296676] Key type .llcrypt registered [ 6086.367801] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6087.096629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6087.114182] alg: No test for adler32 (adler32-zlib) [ 6088.076456] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6088.261160] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6089.887412] Key type lgssc registered [ 6090.722904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6098.182555] loop0: detected capacity change from 0 to 204800000 [ 6098.321786] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6098.327834] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6098.564692] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6099.252864] loop0: detected capacity change from 0 to 204800000 [ 6099.377407] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6099.387021] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6099.652795] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6100.272909] loop0: detected capacity change from 0 to 204800000 [ 6100.396834] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6100.406557] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6100.476275] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6101.132672] loop0: detected capacity change from 0 to 204800000 [ 6101.256092] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6101.900738] loop0: detected capacity change from 0 to 204800000 [ 6101.915423] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6104.090208] loop0: detected capacity change from 0 to 204800000 [ 6104.106031] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6106.472426] loop0: detected capacity change from 0 to 204800000 [ 6106.491353] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6108.884153] loop0: detected capacity change from 0 to 204800000 [ 6108.924278] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6111.440091] loop0: detected capacity change from 0 to 204800000 [ 6111.519750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.541637] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.587691] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6111.597769] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6111.602646] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6111.609358] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6111.615211] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6111.620943] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6111.626207] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6111.637315] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6111.643381] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6111.650981] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6111.657898] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6111.662481] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6111.667237] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6111.671593] Lustre: 221460:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6115.420062] Lustre: server umount MGS complete [ 6115.948828] loop0: detected capacity change from 0 to 204800000 [ 6115.994370] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.244169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6116.270047] Lustre: lustre-MDT0000: Aborting client recovery [ 6116.827374] loop1: detected capacity change from 0 to 204800000 [ 6116.917077] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.977474] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6116.991608] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6117.006145] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6117.018423] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6117.024775] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6117.032153] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6117.045651] Lustre: 221893:0:(osd_scrub.c:1829:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6117.055816] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6117.173085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.210215] Lustre: lustre-MDT0001: Aborting client recovery [ 6117.713154] loop2: detected capacity change from 0 to 204800000 [ 6117.842984] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6117.886564] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6117.895138] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6117.902252] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6117.909758] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6117.914955] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6117.929187] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6117.938601] Lustre: 221972:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6118.031235] Lustre: Found index 0 for lustre-OST0000, updating log [ 6118.095852] Lustre: lustre-OST0000: Aborting recovery [ 6118.731335] loop3: detected capacity change from 0 to 204800000 [ 6118.856811] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.912581] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6118.924961] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6118.931670] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6118.945560] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6118.960680] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6118.968796] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6118.978462] Lustre: 222056:0:(osd_scrub.c:1829:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6119.056448] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6119.059772] Lustre: Skipped 1 previous similar message [ 6119.069640] Lustre: lustre-OST0001: Aborting recovery [ 6124.529089] LustreError: 221837:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -108 [ 6126.572839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 6126.581680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 6126.614800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 6126.628129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 6127.595362] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 6127.597731] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 6131.565037] LustreError: 221804:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 461us: rc = -2 [ 6148.064578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.065936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6148.066313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6148.073472] Lustre: Skipped 2 previous similar messages [ 6149.445151] Lustre: server umount lustre-MDT0000 complete [ 6149.865967] LustreError: 221786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222176 with bad export cookie 9585526893518745688 [ 6149.867040] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.870590] LustreError: 221786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6153.185266] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.191990] LustreError: 222410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.192261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6153.192265] Lustre: Skipped 3 previous similar messages [ 6153.200125] Lustre: Skipped 2 previous similar messages [ 6153.215161] LustreError: 222410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6154.208156] LustreError: 221803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.208773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6154.214358] LustreError: 221803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6154.219255] Lustre: Skipped 2 previous similar messages [ 6156.143380] Lustre: server umount lustre-MDT0001 complete [ 6162.740146] Lustre: server umount lustre-OST0000 complete [ 6169.415145] Lustre: server umount lustre-OST0001 complete [ 6174.001192] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 09:16:39 (1775222199) [ 6188.992876] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6194.541359] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6214.513929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.651216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6214.670394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6214.729209] Lustre: lustre-MDT0000: new disk, initializing [ 6214.786820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.798982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6216.741881] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6222.865176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.923384] Lustre: 226098:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6222.963886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6222.966724] Lustre: Skipped 1 previous similar message [ 6223.008242] Lustre: lustre-MDT0001: new disk, initializing [ 6223.067257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6223.090667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6223.100975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6225.045952] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6229.539905] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6231.370644] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6234.642748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.779808] Lustre: lustre-OST0000: new disk, initializing [ 6234.782384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6234.819365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6237.254524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6237.258935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6237.290865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6237.344502] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6241.329863] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6243.945801] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 6245.448266] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 6254.505222] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 6259.143238] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 6303.728995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.737640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6303.741496] Lustre: Skipped 1 previous similar message [ 6308.839813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.851253] Lustre: Skipped 1 previous similar message [ 6314.975147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6315.172319] Lustre: server umount lustre-OST0000 complete [ 6324.195651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.233805] Lustre: Skipped 1 previous similar message [ 6324.244126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.246913] Lustre: Skipped 2 previous similar messages [ 6327.810489] Lustre: server umount lustre-MDT0000 complete [ 6329.312067] LustreError: 226109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.356268] LustreError: 226109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6333.591835] LustreError: 226876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222359 with bad export cookie 9585526893518748397 [ 6333.608878] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.618660] LustreError: 226876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6334.523769] Lustre: server umount lustre-MDT0001 complete [ 6345.357580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.054437] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6354.292376] Lustre: server umount MGS complete [ 6363.404324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.946452] LustreError: 231573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.967520] LustreError: 231586:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6364.050822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.549349] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6369.249922] LustreError: 231574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.259979] LustreError: 231574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6374.673464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.731035] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6383.430988] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6386.491820] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6392.066229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.902937] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6401.968897] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6435.297037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6435.301433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.312494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6435.322817] Lustre: Skipped 2 previous similar messages [ 6435.332087] Lustre: Skipped 2 previous similar messages [ 6439.587541] Lustre: server umount lustre-OST0000 complete [ 6445.537745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.549425] Lustre: Skipped 1 previous similar message [ 6457.313554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6457.570130] Lustre: server umount lustre-MDT0000 complete [ 6459.876165] LustreError: 231569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.885035] LustreError: 231569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6460.372792] LustreError: 231553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222486 with bad export cookie 9585526893518750182 [ 6460.375787] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6460.383545] LustreError: 231553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6460.638988] Lustre: server umount lustre-MDT0001 complete [ 6468.800434] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6470.986703] Key type lgssc unregistered [ 6471.258296] LNet: 235184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.267271] LNetError: 235184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6471.283408] LNet: Removed LNI 192.168.201.109@tcp [ 6471.909797] Key type .llcrypt unregistered [ 6471.911523] Key type ._llcrypt unregistered [ 6499.153799] Key type ._llcrypt registered [ 6499.156386] Key type .llcrypt registered [ 6499.260845] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6510.407632] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6511.561343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6511.590858] alg: No test for adler32 (adler32-zlib) [ 6512.625701] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6512.838554] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6514.463148] Key type lgssc registered [ 6515.378596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6546.279398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6546.320171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.494625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6547.522486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6547.588654] Lustre: lustre-MDT0000: new disk, initializing [ 6547.657974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.677125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6551.124625] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6559.492328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.608157] Lustre: 239083:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6559.673435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6559.677088] Lustre: Skipped 1 previous similar message [ 6559.728367] Lustre: lustre-MDT0001: new disk, initializing [ 6559.800753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6559.841699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6559.855192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6563.062141] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6568.934222] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6571.731713] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6577.102746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.379908] Lustre: lustre-OST0000: new disk, initializing [ 6577.383724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6577.452717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.312402] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6586.961426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6586.973333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6587.119867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6589.394687] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6595.999328] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 6603.743476] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 6608.702327] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 6612.879950] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 6653.921849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6653.926824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.947915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.958307] Lustre: Skipped 1 previous similar message [ 6659.044833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6659.056499] Lustre: Skipped 1 previous similar message [ 6664.168806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.179296] Lustre: Skipped 1 previous similar message [ 6666.207507] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6666.395745] Lustre: server umount lustre-OST0000 complete [ 6672.352238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6672.357801] LustreError: Skipped 1 previous similar message [ 6672.370033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.380863] Lustre: Skipped 1 previous similar message [ 6672.384827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6674.403179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.043289] Lustre: server umount lustre-MDT0000 complete [ 6679.581842] LustreError: 239862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222705 with bad export cookie 18271691699306728670 [ 6679.583177] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.602364] LustreError: 239862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6679.957805] Lustre: server umount lustre-MDT0001 complete [ 6687.464329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.816842] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6695.778687] Lustre: server umount MGS complete [ 6704.856708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.165583] LustreError: 244514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.182665] LustreError: 244514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6705.189685] LustreError: 244527:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6705.243797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.315193] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6710.240677] LustreError: 244515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.366462] LustreError: 244514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.256376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.580703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6721.683765] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6726.624802] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6729.906509] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6736.156481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6736.436518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.923422] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6746.781874] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6778.337023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6778.340927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.348315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6778.352082] Lustre: Skipped 2 previous similar messages [ 6781.409635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6784.541209] Lustre: server umount lustre-OST0000 complete [ 6789.094296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6789.101882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.109778] Lustre: Skipped 1 previous similar message [ 6789.119421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6789.125215] Lustre: Skipped 2 previous similar messages [ 6802.399133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6802.632844] Lustre: server umount lustre-MDT0000 complete [ 6805.891166] LustreError: 246903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775222832 with bad export cookie 18271691699306730511 [ 6805.894833] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6805.902277] LustreError: 246903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6806.257667] Lustre: server umount lustre-MDT0001 complete [ 6816.987446] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6819.658592] Key type lgssc unregistered [ 6819.884755] LNet: 248104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6819.892455] LNetError: 248104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6819.906550] LNet: Removed LNI 192.168.201.109@tcp [ 6820.659173] Key type .llcrypt unregistered [ 6820.665509] Key type ._llcrypt unregistered [ 6840.099389] Key type ._llcrypt registered [ 6840.100789] Key type .llcrypt registered [ 6840.188201] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6852.389460] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6853.447204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6853.587790] alg: No test for adler32 (adler32-zlib) [ 6854.689179] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 6854.971097] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6856.657121] Key type lgssc registered [ 6857.862885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6879.645552] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6893.088911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6893.107874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.337668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6894.370446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6894.438315] Lustre: lustre-MDT0000: new disk, initializing [ 6894.530457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.555398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6898.699250] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6907.511058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6907.746235] Lustre: lustre-OST0000: new disk, initializing [ 6907.751654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6907.760949] Lustre: Skipped 1 previous similar message [ 6907.828123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.332778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6910.340977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6910.373970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6912.242930] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7632.616557] Lustre: server umount lustre-MDT0000 complete [ 7636.502365] LustreError: 251272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223662 with bad export cookie 7681739080495724490 [ 7636.508635] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7646.938629] Lustre: server umount lustre-OST0000 complete [ 7674.748805] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 7685.761474] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7724.010644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.241866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7724.278574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7724.355365] Lustre: lustre-MDT0000: new disk, initializing [ 7724.524628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.549918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7729.045409] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7739.513878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.669548] Lustre: 256333:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7739.759298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7739.762209] Lustre: Skipped 1 previous similar message [ 7739.847487] Lustre: lustre-MDT0001: new disk, initializing [ 7739.964423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7740.009637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7740.028689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7744.947223] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7751.625290] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7754.871944] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7760.855137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7761.216510] Lustre: lustre-OST0000: new disk, initializing [ 7761.220121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7761.332533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7766.870765] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7769.170325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7769.176454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7769.265660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7774.140760] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7778.860476] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7779.020613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7783.984241] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7784.223829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7790.058133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.074689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7790.080679] Lustre: Skipped 1 previous similar message [ 7790.081654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.101152] Lustre: Skipped 1 previous similar message [ 7792.468521] Lustre: server umount lustre-OST0000 complete [ 7800.289974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.300490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.312216] Lustre: Skipped 1 previous similar message [ 7800.337650] Lustre: Skipped 1 previous similar message [ 7802.766619] Lustre: server umount lustre-MDT0000 complete [ 7806.623195] LustreError: 258071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223832 with bad export cookie 7681739080498665309 [ 7806.624882] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7806.641039] LustreError: 258071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7807.247618] Lustre: server umount lustre-MDT0001 complete [ 7811.330515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 7813.218834] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 09:43:57 (1775223837) [ 7820.055573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.548834] LustreError: 260240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.572843] LustreError: 260240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7820.618904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7824.964926] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7825.890045] LustreError: 260241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.993966] LustreError: 260240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.757993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.138784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.134307] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7845.598498] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7849.877284] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7856.525469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.785962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7862.502903] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7868.401757] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7875.730742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7876.002185] Lustre: lustre-OST0001: new disk, initializing [ 7876.015968] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7876.153508] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.112491] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7883.358585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7883.368189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7883.407731] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7884.454882] LustreError: 263156:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7884.471708] LustreError: 263156:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7885.475991] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7887.522922] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7887.538777] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 7891.746151] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7891.754265] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 7892.182462] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7892.462443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7896.332894] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff998406248800.ost_server_uuid 50 [ 7897.794436] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff998406248800.ost_server_uuid in FULL state after 0 sec [ 7899.940672] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7899.962873] LustreError: 263151:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [ 7902.038206] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff998406248800.ost_server_uuid 50 [ 7903.868011] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff998406248800.ost_server_uuid in FULL state after 0 sec [ 7911.405849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7919.599211] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.618080] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7923.031552] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 7923.036067] Lustre: Skipped 1 previous similar message [ 7925.681967] Lustre: server umount lustre-OST0001 complete [ 7928.128179] LustreError: 261835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.887155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7931.906298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.929605] Lustre: Skipped 1 previous similar message [ 7931.935218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.941986] Lustre: Skipped 2 previous similar messages [ 7934.954835] LustreError: 261835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.987969] LustreError: 261835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7937.225957] Lustre: server umount lustre-OST0000 complete [ 7941.603994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7941.618065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.630046] Lustre: Skipped 1 previous similar message [ 7941.637181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7941.643015] Lustre: Skipped 2 previous similar messages [ 7947.085768] Lustre: server umount lustre-MDT0000 complete [ 7950.313582] LustreError: 260241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.336817] LustreError: 260241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7950.806602] LustreError: 260221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775223977 with bad export cookie 7681739080498666191 [ 7950.809196] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7950.819075] LustreError: 260221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7951.203525] Lustre: server umount lustre-MDT0001 complete [ 7961.844369] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7964.606500] Key type lgssc unregistered [ 7964.899786] LNet: 265182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7964.920992] LNetError: 265182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7965.991934] LNet: Removed LNI 192.168.201.109@tcp [ 7966.925922] Key type .llcrypt unregistered [ 7966.927453] Key type ._llcrypt unregistered [ 7987.058524] Key type ._llcrypt registered [ 7987.061851] Key type .llcrypt registered [ 7987.164173] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7988.386954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7988.410549] alg: No test for adler32 (adler32-zlib) [ 7989.449374] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 7989.685730] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7991.359192] Key type lgssc registered [ 7992.390681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7999.991654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8000.028106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.543043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.495912] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8010.885829] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8011.749435] LustreError: 266566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8011.781198] LustreError: 266566:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8016.867273] LustreError: 266567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.265440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.655053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8021.809819] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8029.999654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.282913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8036.126469] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8041.663953] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8042.534384] LustreError: 268115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.593104] LustreError: 268115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.068771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8048.393844] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8049.476024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8049.481046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 8053.553234] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8058.512734] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8064.350078] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8064.571888] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8157.151859] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8157.154378] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.159503] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8157.167206] Lustre: Skipped 1 previous similar message [ 8162.282745] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8162.286208] Lustre: Skipped 2 previous similar messages [ 8163.181605] Lustre: server umount lustre-OST0001 complete [ 8167.392701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8167.397114] LustreError: 268118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8167.398476] LustreError: Skipped 1 previous similar message [ 8167.398500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.410820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.429670] LustreError: 268118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8167.431527] Lustre: Skipped 1 previous similar message [ 8172.516374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8172.529971] Lustre: Skipped 1 previous similar message [ 8173.311095] Lustre: server umount lustre-OST0000 complete [ 8176.612456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8176.618628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.626014] Lustre: Skipped 1 previous similar message [ 8176.630802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.617209] Lustre: server umount lustre-MDT0000 complete [ 8182.752291] LustreError: 267376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.763531] LustreError: 267376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8185.158273] LustreError: 266548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224211 with bad export cookie 11765051632061302232 [ 8185.159667] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8185.168697] LustreError: 266548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8185.384352] Lustre: server umount lustre-MDT0001 complete [ 8192.136558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8193.875442] Key type lgssc unregistered [ 8194.090403] LNet: 271134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8194.097907] LNetError: 271134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8194.114853] LNet: Removed LNI 192.168.201.109@tcp [ 8194.595682] Key type .llcrypt unregistered [ 8194.597234] Key type ._llcrypt unregistered [ 8228.549416] Key type ._llcrypt registered [ 8228.554835] Key type .llcrypt registered [ 8228.632691] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8229.658937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8229.677141] alg: No test for adler32 (adler32-zlib) [ 8230.656241] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8230.889677] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8232.551385] Key type lgssc registered [ 8233.384051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8240.196601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8240.212747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.536547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8245.338028] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8250.850526] LustreError: 273384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.867790] LustreError: 273384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8252.458437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.829664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8256.152984] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8258.471288] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8263.371836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.798787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.841406] LustreError: 274840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8265.856077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 8268.449724] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8271.335564] LustreError: 274840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.448543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8275.638445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.685254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 8280.262945] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8288.382481] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8293.131795] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8308.719130] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.725750] Lustre: Skipped 1 previous similar message [ 8308.732243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8308.736318] Lustre: Skipped 2 previous similar messages [ 8313.039628] Lustre: server umount lustre-MDT0000 complete [ 8313.826257] LustreError: 274859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8313.846880] LustreError: 274859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8315.699821] LustreError: 273366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224341 with bad export cookie 15183597877631758917 [ 8315.701360] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8315.706361] LustreError: 273366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8315.949255] Lustre: server umount lustre-MDT0001 complete [ 8329.745030] Lustre: server umount lustre-OST0000 complete [ 8342.800366] Lustre: server umount lustre-OST0001 complete [ 8355.440631] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8363.243917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.533325] LustreError: 278570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.619685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.792715] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8373.224623] LustreError: 278571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.248916] LustreError: 278571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8373.284731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.550964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8376.958509] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8378.981830] Lustre: 279675:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8383.983285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.347456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 8389.296698] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8393.707771] LustreError: 280320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.746357] LustreError: 280320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8395.939266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8396.169735] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8396.175486] Lustre: Skipped 1 previous similar message [ 8398.223097] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 8401.266611] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8407.421203] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8410.240731] Lustre: 281523:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8419.813825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8419.822521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.830415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8419.845290] Lustre: Skipped 5 previous similar messages [ 8419.849025] Lustre: Skipped 2 previous similar messages [ 8426.504899] Lustre: server umount lustre-MDT0000 complete [ 8428.991075] LustreError: 278550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224455 with bad export cookie 15183597877631760618 [ 8428.992040] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8429.002505] LustreError: 278550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8429.202910] Lustre: server umount lustre-MDT0001 complete [ 8442.157444] Lustre: server umount lustre-OST0000 complete [ 8455.189646] Lustre: server umount lustre-OST0001 complete [ 8457.858610] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 8459.137986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 8460.599812] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 09:54:45 (1775224485) [ 8477.049815] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8483.402551] loop0: detected capacity change from 0 to 1099511627776 [ 8483.765549] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 8483.773986] *** VALIDATE xfs *** [ 8483.782392] XFS (loop0): Mounting V5 Filesystem [ 8483.796813] XFS (loop0): Ending clean mount [ 8484.708461] loop1: detected capacity change from 0 to 18691697672192 [ 8554.279185] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 8561.989531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8588.153947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.356368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8588.390482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8588.473854] Lustre: lustre-MDT0000: new disk, initializing [ 8588.565178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.579187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8592.319071] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8600.405202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.535582] Lustre: 286598:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8600.584435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8600.588610] Lustre: Skipped 1 previous similar message [ 8600.664416] Lustre: lustre-MDT0001: new disk, initializing [ 8600.744261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8600.757542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8604.337970] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8610.025232] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8612.550158] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8617.379923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8617.635597] Lustre: lustre-OST0000: new disk, initializing [ 8617.641197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8619.261681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8619.269242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8619.354213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8622.296469] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8628.031052] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8631.236742] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8631.380119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8634.717972] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8634.879443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8639.974881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.978575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8639.988419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8639.988427] Lustre: Skipped 1 previous similar message [ 8639.988646] Lustre: Skipped 1 previous similar message [ 8639.993189] LustreError: Skipped 1 previous similar message [ 8642.451266] Lustre: server umount lustre-OST0000 complete [ 8645.089399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.103178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.111124] Lustre: Skipped 2 previous similar messages [ 8650.208616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8650.216579] Lustre: Skipped 1 previous similar message [ 8651.102281] Lustre: server umount lustre-MDT0000 complete [ 8653.536403] LustreError: 286589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224679 with bad export cookie 15183597877631762172 [ 8653.546240] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8653.732682] Lustre: server umount lustre-MDT0001 complete [ 8659.030494] XFS (loop0): Unmounting Filesystem [ 8662.268520] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 09:58:07 (1775224687) [ 8666.905213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.157594] LustreError: 290749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.168755] LustreError: 290749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8667.207303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.210113] Lustre: Skipped 2 previous similar messages [ 8670.180385] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8676.195338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.716461] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8683.749898] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8686.323484] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8691.034071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8694.688423] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8698.016767] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8711.652310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8711.653957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.658866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8711.671678] Lustre: Skipped 2 previous similar messages [ 8711.691380] Lustre: Skipped 1 previous similar message [ 8721.890598] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8722.037956] Lustre: server umount lustre-OST0000 complete [ 8727.014367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8732.130428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8732.142839] Lustre: Skipped 7 previous similar messages [ 8739.295184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8739.445779] Lustre: server umount lustre-MDT0000 complete [ 8742.044102] LustreError: 290731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775224768 with bad export cookie 15183597877631763054 [ 8742.046067] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8742.051922] LustreError: 290731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8742.330093] Lustre: server umount lustre-MDT0001 complete [ 8750.429890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8752.417397] Key type lgssc unregistered [ 8752.619729] LNet: 294028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8752.623135] LNetError: 294028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8752.643133] LNet: Removed LNI 192.168.201.109@tcp [ 8753.277169] Key type .llcrypt unregistered [ 8753.279835] Key type ._llcrypt unregistered [ 8772.302715] Key type ._llcrypt registered [ 8772.304467] Key type .llcrypt registered [ 8772.379142] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8773.153421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8773.174722] alg: No test for adler32 (adler32-zlib) [ 8774.087200] Lustre: Lustre: Build Version: 2.17.51_74_g8e7ba6b [ 8774.265568] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8775.879222] Key type lgssc registered [ 8776.977511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.320902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8782.335049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.652129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.404255] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8792.163293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.480335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8795.098859] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8798.788433] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8801.301468] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8805.834408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8806.107075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8810.372867] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8814.133305] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8824.288331] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8824.293226] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.306813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8827.216946] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 8827.879539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.297778] Lustre: server umount lustre-MDT0001 complete [ 8832.317626] LustreError: 297327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8832.324261] LustreError: 297327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8832.994151] LustreError: 297327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8833.012215] LustreError: 297327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8837.444866] LustreError: 295551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.732088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.813489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8837.820198] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8837.837937] Lustre: 295546:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8838.025952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.664880] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8843.241411] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8845.293307] LustreError: 298826:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8845.297296] LustreError: 298826:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8845.303279] LustreError: 295548:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8845.354139] LustreError: 298829:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -17 [ 8845.367898] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 8845.367898] [ 9147.739053] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 9147.743633] Lustre: Skipped 2 previous similar messages [ 9150.434363] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9150.439692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.448708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9150.463820] Lustre: Skipped 1 previous similar message [ 9152.520565] Lustre: server umount lustre-MDT0001 complete [ 9152.832672] LustreError: 301598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.953755] LustreError: 304619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.975223] LustreError: 304619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9161.123736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.173108] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9161.176261] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9161.195653] Lustre: 295546:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9161.393962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.834586] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9166.833966] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9169.926963] LustreError: 307642:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9169.951576] LustreError: 307642:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9169.960481] LustreError: 295548:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9169.965635] LustreError: 295548:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9170.048591] LustreError: 307647:0:(obd_config.c:760:class_setup()) Device 8 already setup (type osp) [ 9170.052495] LustreError: 307647:0:(obd_config.c:864:class_detach()) OBD device 8 still set up [ 9170.055861] LustreError: 307647:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -17 [ 9170.066715] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.109@tcp [ 9170.066715] [ 9468.898679] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9468.901343] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.903617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9468.903624] Lustre: Skipped 1 previous similar message [ 9468.914872] Lustre: Skipped 1 previous similar message [ 9474.665647] Lustre: server umount lustre-MDT0001 complete [ 9478.456766] LustreError: 301598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9478.468946] LustreError: 301598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9479.142065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9479.194067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9479.197134] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9479.217309] Lustre: 295546:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9479.365091] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 9481.126067] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9484.776474] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9488.869340] LustreError: 318095:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9488.872927] LustreError: 318095:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9488.876411] LustreError: 295548:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9488.879101] LustreError: 295548:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9488.902897] LustreError: 318098:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 9488.912459] LustreError: 318098:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.201.109@tcp: cfg command failed: rc = -114 [ 9488.919065] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 9488.919065] [ 9797.090883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.091249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9797.092344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9797.092349] Lustre: Skipped 4 previous similar messages [ 9797.097311] Lustre: Skipped 1 previous similar message [ 9802.432969] Lustre: server umount lustre-MDT0000 complete [ 9804.216658] LustreError: 307226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225830 with bad export cookie 16587863282849276055 [ 9804.219567] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9804.223548] LustreError: 307226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9807.332867] LustreError: 304619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.333293] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.340150] LustreError: 304619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9807.348899] Lustre: Skipped 1 previous similar message [ 9810.448841] Lustre: server umount lustre-MDT0001 complete [ 9812.347617] Lustre: server umount lustre-OST0000 complete [ 9816.275116] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9820.213498] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9834.082251] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 10:17:39 (1775225859) [ 9837.072423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.185402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9837.200918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9837.241375] Lustre: lustre-MDT0000: new disk, initializing [ 9837.277861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.286108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9838.856235] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9843.883797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.942258] Lustre: 331975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9843.967200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9843.969540] Lustre: Skipped 1 previous similar message [ 9844.004870] Lustre: lustre-MDT0001: new disk, initializing [ 9844.043023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.064772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9844.077623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9845.712322] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9849.599540] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9851.077435] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9853.969461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.096959] Lustre: lustre-OST0000: new disk, initializing [ 9854.099774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9854.141230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.883339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9855.887437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9855.934083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9856.501534] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9860.056533] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9864.671935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9864.676373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.681231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9864.683796] Lustre: Skipped 7 previous similar messages [ 9869.224300] Lustre: server umount lustre-MDT0000 complete [ 9870.653215] LustreError: 331967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775225896 with bad export cookie 16587863282849278008 [ 9870.655861] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9870.656176] LustreError: 332763:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 9870.658484] LustreError: 331967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9870.806590] Lustre: server umount lustre-MDT0001 complete [ 9882.530551] Lustre: server umount lustre-OST0000 complete [ 9892.686520] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9896.336181] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9908.425457] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9625 sec ========== 10:18:53 (1775225933) [ 9909.102109] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:18:54 (1775225934) === [ 9910.323258] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:18:55 (1775225935) === [ 9925.380305] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9926.529126] Key type lgssc unregistered [ 9926.671561] LNet: 339903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9926.674432] LNetError: 339903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9926.681341] LNet: Removed LNI 192.168.201.109@tcp [ 9927.018470] Key type .llcrypt unregistered [ 9927.019898] Key type ._llcrypt unregistered