[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507932702 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003137] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007996] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.022673] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027722] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030038] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031146] *** VALIDATE tmpfs *** [ 0.033246] *** VALIDATE proc *** [ 0.034291] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039035] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043422] debug: unmapping init [mem 0xffffffffa9c59000-0xffffffffa9c60fff] [ 0.045198] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046722] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054347] rcu: Hierarchical SRCU implementation. [ 0.056686] smp: Bringing up secondary CPUs ... [ 0.057591] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.062016] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143378] node 0 deferred pages initialised in 73ms [ 0.146182] devtmpfs: initialized [ 0.147364] x86/mm: Memory block size: 128MB [ 0.150046] gcov: version magic: 0x41383552 [ 0.152309] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154305] pinctrl core: initialized pinctrl subsystem [ 0.155225] [ 0.155803] ************************************************************* [ 0.156013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157016] ** ** [ 0.158014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159015] ** ** [ 0.160016] ** This means that this kernel is built to expose internal ** [ 0.161017] ** IOMMU data structures, which may compromise security on ** [ 0.162015] ** your system. ** [ 0.163014] ** ** [ 0.164017] ** If you see this message and you are not debugging the ** [ 0.165013] ** kernel, report this immediately to your vendor! ** [ 0.166015] ** ** [ 0.167016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168015] ************************************************************* [ 0.169753] NET: Registered protocol family 16 [ 0.170490] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173777] cpuidle: using governor menu [ 0.176432] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177547] PCI: Using configuration type 1 for base access [ 0.178133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188051] cryptd: max_cpu_qlen set to 1000 [ 0.190256] ACPI: Added _OSI(Module Device) [ 0.191015] ACPI: Added _OSI(Processor Device) [ 0.192024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197153] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201089] ACPI: Interpreter enabled [ 0.202069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203017] ACPI: Using IOAPIC for interrupt routing [ 0.204115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205427] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214374] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219292] acpiphp: Slot [2] registered [ 0.220106] acpiphp: Slot [5] registered [ 0.221030] acpiphp: Slot [6] registered [ 0.222092] acpiphp: Slot [7] registered [ 0.223107] acpiphp: Slot [8] registered [ 0.224120] acpiphp: Slot [9] registered [ 0.225102] acpiphp: Slot [10] registered [ 0.225893] acpiphp: Slot [3] registered [ 0.226050] acpiphp: Slot [4] registered [ 0.226966] acpiphp: Slot [11] registered [ 0.227084] acpiphp: Slot [12] registered [ 0.228110] acpiphp: Slot [13] registered [ 0.229168] acpiphp: Slot [14] registered [ 0.230102] acpiphp: Slot [15] registered [ 0.231103] acpiphp: Slot [16] registered [ 0.232085] acpiphp: Slot [17] registered [ 0.233105] acpiphp: Slot [18] registered [ 0.234133] acpiphp: Slot [19] registered [ 0.235110] acpiphp: Slot [20] registered [ 0.236134] acpiphp: Slot [21] registered [ 0.237111] acpiphp: Slot [22] registered [ 0.238132] acpiphp: Slot [23] registered [ 0.239131] acpiphp: Slot [24] registered [ 0.240115] acpiphp: Slot [25] registered [ 0.241127] acpiphp: Slot [26] registered [ 0.242117] acpiphp: Slot [27] registered [ 0.243107] acpiphp: Slot [28] registered [ 0.244099] acpiphp: Slot [29] registered [ 0.245121] acpiphp: Slot [30] registered [ 0.246106] acpiphp: Slot [31] registered [ 0.247067] PCI host bridge to bus 0000:00 [ 0.248021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.249026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.251037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.253028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.254247] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.256071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.258320] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.267503] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.268021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.269020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.270015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.272025] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.273854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.274047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.275806] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.277599] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.283953] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.286017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.290012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.293020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.296017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.304018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.310747] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.313025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.316018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.324017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.331783] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.343022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.351023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.370027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.383598] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.399030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.409024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.428016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.438000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.444020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.449015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.463017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.471973] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.479015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.492017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.523022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.538372] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.541394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.544344] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.546386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.549265] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.555054] iommu: Default domain type: Passthrough [ 0.556518] SCSI subsystem initialized [ 0.558125] ACPI: bus type USB registered [ 0.560121] usbcore: registered new interface driver usbfs [ 0.562082] usbcore: registered new interface driver hub [ 0.564107] usbcore: registered new device driver usb [ 0.565134] pps_core: LinuxPPS API ver. 1 registered [ 0.568020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.571066] PTP clock support registered [ 0.574101] EDAC MC: Ver: 3.0.0 [ 0.576283] PCI: Using ACPI for IRQ routing [ 0.578075] NetLabel: Initializing [ 0.579012] NetLabel: domain hash size = 128 [ 0.580016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.581119] NetLabel: unlabeled traffic allowed by default [ 0.582132] vgaarb: loaded [ 0.583262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.585014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.591000] clocksource: Switched to clocksource kvm-clock [ 0.701368] VFS: Disk quotas dquot_6.6.0 [ 0.702660] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.704539] *** VALIDATE ramfs *** [ 0.705707] *** VALIDATE hugetlbfs *** [ 0.707215] pnp: PnP ACPI init [ 0.709494] pnp: PnP ACPI: found 6 devices [ 0.736396] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.740092] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.742474] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.744789] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.747534] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.750327] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.753615] NET: Registered protocol family 2 [ 0.756823] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.764261] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.768451] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.773892] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.776620] TCP: Hash tables configured (established 65536 bind 65536) [ 0.779044] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.781428] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.783514] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.785802] NET: Registered protocol family 1 [ 0.788458] RPC: Registered named UNIX socket transport module. [ 0.790923] RPC: Registered udp transport module. [ 0.792367] RPC: Registered tcp transport module. [ 0.794226] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.797151] NET: Registered protocol family 44 [ 0.799038] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.801482] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.803729] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.806033] PCI: CLS 0 bytes, default 64 [ 0.807545] Unpacking initramfs... [ 2.218603] debug: unmapping init [mem 0xffff896e3cc54000-0xffff896e3ffbffff] [ 2.222919] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.225519] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.228529] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.721242] Initialise system trusted keyrings [ 2.722945] Key type blacklist registered [ 2.724478] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.735129] zbud: loaded [ 2.738040] *** VALIDATE nfs *** [ 2.739129] *** VALIDATE nfs4 *** [ 2.740517] pstore: using deflate compression [ 2.743753] Platform Keyring initialized [ 2.850407] NET: Registered protocol family 38 [ 2.851858] Key type asymmetric registered [ 2.854783] Asymmetric key parser 'x509' registered [ 2.856359] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.858961] io scheduler mq-deadline registered [ 2.860273] io scheduler kyber registered [ 2.861634] io scheduler bfq registered [ 2.863124] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.865580] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.867860] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.870259] ACPI: Power Button [PWRF] [ 2.874909] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.880741] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.891201] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.897072] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.915556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.942311] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.971509] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.975955] Non-volatile memory driver v1.3 [ 2.977402] Linux agpgart interface v0.103 [ 3.007045] virtio_blk virtio1: [vda] 146856 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.010070] vda: detected capacity change from 0 to 75190272 [ 3.028057] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.030496] vdb: detected capacity change from 0 to 1073741824 [ 3.049954] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.057957] vdc: detected capacity change from 0 to 2621440000 [ 3.089076] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.092570] vdd: detected capacity change from 0 to 2621440000 [ 3.110424] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.112945] vde: detected capacity change from 0 to 4294967296 [ 3.132116] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.135812] vdf: detected capacity change from 0 to 4294967296 [ 3.144982] libphy: Fixed MDIO Bus: probed [ 3.152936] usbcore: registered new interface driver usbserial_generic [ 3.154928] usbserial: USB Serial support registered for generic [ 3.157642] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.163145] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.165635] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.168805] mousedev: PS/2 mouse device common for all mice [ 3.171835] rtc_cmos 00:05: RTC can wake from S4 [ 3.175219] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.175340] rtc_cmos 00:05: registered as rtc0 [ 3.181032] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.182671] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.184498] intel_pstate: CPU model not supported [ 3.190405] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.195656] hid: raw HID events driver (C) Jiri Kosina [ 3.197868] usbcore: registered new interface driver usbhid [ 3.199603] usbhid: USB HID core driver [ 3.201399] drop_monitor: Initializing network drop monitor service [ 3.204195] Initializing XFRM netlink socket [ 3.206592] NET: Registered protocol family 10 [ 3.210034] Segment Routing with IPv6 [ 3.211756] NET: Registered protocol family 17 [ 3.214109] mpls_gso: MPLS GSO support [ 3.220102] RAS: Correctable Errors collector initialized. [ 3.222322] AVX version of gcm_enc/dec engaged. [ 3.224121] AES CTR mode by8 optimization enabled [ 3.312479] sched_clock: Marking stable (3312456917, 0)->(4407808559, -1095351642) [ 3.315895] registered taskstats version 1 [ 3.318199] Loading compiled-in X.509 certificates [ 3.319786] zswap: loaded using pool lzo/zbud [ 3.343971] Key type big_key registered [ 3.355195] Key type encrypted registered [ 3.356854] ima: No TPM chip found, activating TPM-bypass! [ 3.358318] ima: Allocated hash algorithm: sha1 [ 3.360168] ima: No architecture policies found [ 3.362018] evm: Initialising EVM extended attributes: [ 3.363740] evm: security.selinux [ 3.365044] evm: security.ima [ 3.366214] evm: security.capability [ 3.367578] evm: HMAC attrs: 0x1 [ 3.369913] rtc_cmos 00:05: setting system clock to 2026-08-27 18:43:31 UTC (1787856211) [ 3.376371] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 3.379292] debug: unmapping init [mem 0xffffffffa9982000-0xffffffffa9c58fff] [ 3.390091] Write protecting the kernel read-only data: 28672k [ 3.393501] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 3.396600] debug: unmapping init [mem 0xffffffffa8914000-0xffffffffa89fffff] [ 3.429986] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.438554] systemd[1]: Detected virtualization kvm. [ 3.440952] systemd[1]: Detected architecture x86-64. [ 3.443199] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.470516] systemd[1]: No hostname configured. [ 3.471926] systemd[1]: Set hostname to . [ 3.473632] random: systemd: uninitialized urandom read (16 bytes read) [ 3.475525] systemd[1]: Initializing machine ID from random generator. [ 3.603150] random: systemd: uninitialized urandom read (16 bytes read) [ 3.606836] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.612251] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615340] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.619839] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.256584] device-mapper: uevent: version 1.0.3 [ 4.259337] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.047243] virtio_net virtio0 ens2: renamed from eth0 [ 5.053791] random: fast init done [ 5.106508] scsi host0: ata_piix [ 5.128071] scsi host1: ata_piix [ 5.130339] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.133470] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.511797] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.872403] random: crng init done [ 9.874097] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.376479] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.541675] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.803668] SELinux: Disabled at runtime. [ 11.873898] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.881848] systemd[1]: Detected virtualization kvm. [ 11.883731] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.397307] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.400978] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.406135] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.408885] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.411170] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.419873] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.427956] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ 12.549457] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt.[ 12.854063] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.149744] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.241800] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.325195] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.346425] EDAC sbridge: Ver: 1.1.2 [ 15.251649] Key type dns_resolver registered [ 15.602989] NFS: Registering the id_resolver key type [ 15.607963] Key type id_resolver registered [ 15.609288] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 42.529674] libcfs: loading out-of-tree module taints kernel. [ 42.553070] Key type ._llcrypt registered [ 42.554767] Key type .llcrypt registered [ 42.607212] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 59.334542] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 61.054129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.067124] alg: No test for adler32 (adler32-zlib) [ 62.519333] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 63.947824] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 65.753292] Key type lgssc registered [ 68.381633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.597214] hrtimer: interrupt took 2815084 ns [ 89.339688] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.807679] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 149.354600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.391980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.677926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.752675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.911900] Lustre: lustre-MDT0000: new disk, initializing [ 150.976094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.996809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.207213] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.121796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.257288] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.297369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.306759] Lustre: Skipped 1 previous similar message [ 172.455831] Lustre: lustre-MDT0001: new disk, initializing [ 172.561941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.609551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.625436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.211906] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.850570] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 196.439384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.891764] Lustre: lustre-OST0000: new disk, initializing [ 196.898308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.913611] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.053682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.922109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.935331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.032746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.529099] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.363138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.465333] Lustre: lustre-OST0001: new disk, initializing [ 223.468871] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.474893] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.535083] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.755445] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.553546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.561232] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.719987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.527172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.029829] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.379810] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.712542] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 271.386242] Lustre: DEBUG MARKER: Client: 2.17.57.81 [ 274.149398] Lustre: DEBUG MARKER: MDS: 2.17.57.81 [ 277.038635] Lustre: DEBUG MARKER: OSS: 2.17.57.81 [ 279.276862] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 14:48:05 EDT 2026 [ 298.653365] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 300.623949] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.974685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.980327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.995250] Lustre: Skipped 2 previous similar messages [ 320.006885] Lustre: Skipped 1 previous similar message [ 324.593453] Lustre: server umount lustre-MDT0000 complete [ 328.615679] LustreError: 9513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856536 with bad export cookie 2186250366072434178 [ 328.626540] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.631505] LustreError: 9513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 330.209292] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.209963] LustreError: 6511:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.210265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.210270] Lustre: Skipped 2 previous similar messages [ 330.231273] Lustre: Skipped 2 previous similar messages [ 330.242125] LustreError: 6511:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 333.219452] LustreError: 12159:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.219949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.242869] Lustre: Skipped 2 previous similar messages [ 333.249426] LustreError: 12159:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 334.948976] Lustre: server umount lustre-MDT0001 complete [ 339.109499] Lustre: server umount lustre-OST0000 complete [ 343.119961] Lustre: server umount lustre-OST0001 complete [ 349.395371] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 358.059932] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 396.629821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.898454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.928959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 397.031727] Lustre: lustre-MDT0000: new disk, initializing [ 397.108792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.125168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.929146] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 411.165143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.342677] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.397514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.400700] Lustre: Skipped 1 previous similar message [ 411.471674] Lustre: lustre-MDT0001: new disk, initializing [ 411.585684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 411.633496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 411.649131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 416.379220] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 426.793785] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 432.626926] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 440.298929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.620716] Lustre: lustre-OST0000: new disk, initializing [ 440.624653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.631922] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.710739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.367454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.375204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.569588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.862160] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 460.859717] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.415536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.663287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.651422] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.853362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.058306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.068142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.080532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.700624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.709054] Lustre: Skipped 2 previous similar messages [ 486.370484] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.490671] Lustre: server umount lustre-OST0000 complete [ 493.542862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.566301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.580606] Lustre: Skipped 1 previous similar message [ 493.596430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.610171] Lustre: Skipped 2 previous similar messages [ 505.315838] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.637996] Lustre: server umount lustre-MDT0000 complete [ 508.970383] LustreError: 15868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856717 with bad export cookie 2186250366072435711 [ 508.981640] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.999722] LustreError: 15868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 509.427690] Lustre: server umount lustre-MDT0001 complete [ 514.376441] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:52:00 (1787856720) [ 516.827623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 518.650665] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:52:05 (1787856725) [ 524.958393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.688198] LustreError: 19453:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.759863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.920505] LustreError: 19454:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.325108] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 535.023706] LustreError: 19453:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.360269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.778265] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 553.582447] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.281209] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.405704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 575.325048] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 583.192859] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 595.428816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.431704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 595.441438] Lustre: Skipped 2 previous similar messages [ 595.462670] Lustre: Skipped 6 previous similar messages [ 600.942697] Lustre: server umount lustre-OST0000 complete [ 605.668266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.688149] Lustre: Skipped 1 previous similar message [ 611.718271] Lustre: server umount lustre-MDT0000 complete [ 615.804182] LustreError: 20188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856823 with bad export cookie 2186250366072436586 [ 615.808985] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.825701] LustreError: 20188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 616.376538] Lustre: server umount lustre-MDT0001 complete [ 628.381197] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 631.118608] Key type lgssc unregistered [ 631.432814] LNet: 22990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.441747] LNetError: 22990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 631.468434] LNet: Removed LNI 192.168.204.154@tcp [ 632.489219] Key type .llcrypt unregistered [ 632.498964] Key type ._llcrypt unregistered [ 658.273528] Key type ._llcrypt registered [ 658.275581] Key type .llcrypt registered [ 658.364361] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 659.861343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.961536] alg: No test for adler32 (adler32-zlib) [ 661.238730] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 661.701813] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 663.496223] Key type lgssc registered [ 664.760336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.802520] loop: module loaded [ 673.916669] loop0: detected capacity change from 0 to 204800000 [ 674.223919] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.237204] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.675034] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.209567] loop0: detected capacity change from 0 to 204800000 [ 681.379601] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.391475] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.407552] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.084250] loop0: detected capacity change from 0 to 204800000 [ 693.055287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.082204] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.256524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 694.311917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 694.400599] Lustre: lustre-MDT0000: new disk, initializing [ 694.511697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.540346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 698.798685] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 709.131046] loop1: detected capacity change from 0 to 204800000 [ 715.912875] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.336385] Lustre: lustre-OST07c6: new disk, initializing [ 716.343212] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 716.349338] Lustre: Skipped 1 previous similar message [ 716.358384] Lustre: 26183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 716.459116] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 723.667692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 723.693968] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 723.774995] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 724.700328] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 729.888977] Lustre: 26699:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 729.900741] Lustre: 26699:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 729.911983] Lustre: 26699:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 729.919073] Lustre: 26699:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 729.926254] Lustre: 26699:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 729.933554] Lustre: 26699:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 745.092882] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 745.098480] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 745.106508] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 745.120671] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 745.130503] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 745.140991] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.270882] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 750.278801] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 750.284717] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.292035] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.299414] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.319897] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.327552] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 750.334235] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.342564] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 750.349771] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.355968] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.365297] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.582254] Lustre: 26699:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 755.591372] Lustre: 26699:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.596307] Lustre: 26699:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 755.609124] Lustre: 26699:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.620581] Lustre: 26699:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.634952] Lustre: 26699:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.648755] Lustre: 26699:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 755.666904] Lustre: 26699:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.676468] Lustre: 26699:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.689926] Lustre: 26699:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.697775] Lustre: 26699:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.709340] Lustre: 26699:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.228431] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 764.237634] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 764.257502] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.268506] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.284941] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.292773] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.300786] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 764.317829] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.323541] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 764.330665] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 764.340755] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 764.350896] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 775.928067] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 775.936829] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 775.946786] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 775.955563] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.963339] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.972795] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.979527] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 775.986864] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 775.998029] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.009916] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 776.019152] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.026675] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 821.245110] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 821.259861] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.294251] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 824.860402] Lustre: server umount lustre-OST07c6 complete [ 836.178839] Lustre: server umount lustre-MDT0000 complete [ 855.380842] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 859.057461] Key type lgssc unregistered [ 859.399332] LNet: 28745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.407448] LNetError: 28745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.422936] LNet: Removed LNI 192.168.204.154@tcp [ 860.512777] Key type .llcrypt unregistered [ 860.517322] Key type ._llcrypt unregistered [ 879.703219] Key type ._llcrypt registered [ 879.705575] Key type .llcrypt registered [ 879.816426] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 881.072210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.117132] alg: No test for adler32 (adler32-zlib) [ 882.303666] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 882.742842] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 884.499094] Key type lgssc registered [ 885.870545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.293730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.304042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.841723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.896583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 905.192731] LustreError: 30017:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.211554] LustreError: 30017:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 909.496503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.826372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.791337] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 921.334805] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 929.252105] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 937.973802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.446249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.547871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 945.418716] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 954.209880] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.202465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.207627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.210891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.240544] Lustre: Skipped 1 previous similar message [ 974.310642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.322053] Lustre: Skipped 2 previous similar messages [ 975.044396] Lustre: server umount lustre-OST0000 complete [ 979.425553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.446047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.453183] Lustre: Skipped 1 previous similar message [ 984.548192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.557925] Lustre: Skipped 1 previous similar message [ 989.666384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.679297] Lustre: Skipped 1 previous similar message [ 993.248168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 993.429790] Lustre: server umount lustre-MDT0000 complete [ 994.793499] LustreError: 30014:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.819701] LustreError: 30014:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 997.928602] LustreError: 30754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857206 with bad export cookie 17994164571743919166 [ 997.931041] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.936227] LustreError: 30754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 998.204862] Lustre: server umount lustre-MDT0001 complete [ 1011.550188] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1014.853750] Key type lgssc unregistered [ 1015.287150] LNet: 33762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.299620] LNetError: 33762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.334153] LNet: Removed LNI 192.168.204.154@tcp [ 1016.435292] Key type .llcrypt unregistered [ 1016.437687] Key type ._llcrypt unregistered [ 1035.002201] Key type ._llcrypt registered [ 1035.007313] Key type .llcrypt registered [ 1035.174928] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1036.519486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1036.737893] alg: No test for adler32 (adler32-zlib) [ 1037.967531] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1038.257238] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1039.952278] Key type lgssc registered [ 1041.308614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.660510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1049.683258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.080679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.240229] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1061.366866] LustreError: 35033:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.384740] LustreError: 35033:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1064.971678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.302863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.315355] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1076.744407] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1083.000955] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1090.136199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.452251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.636581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1096.899080] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1105.428775] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1122.280911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.299612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.320807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.349644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.350963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.367793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.381706] Lustre: Skipped 1 previous similar message [ 1126.569501] Lustre: server umount lustre-OST0000 complete [ 1130.465634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.482731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.489350] Lustre: Skipped 1 previous similar message [ 1135.590512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.605159] Lustre: Skipped 1 previous similar message [ 1136.719688] Lustre: server umount lustre-MDT0000 complete [ 1140.705792] LustreError: 35028:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.735269] LustreError: 35028:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1141.459822] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857349 with bad export cookie 11586892513794969582 [ 1141.468490] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1141.480980] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1141.848490] Lustre: server umount lustre-MDT0001 complete [ 1154.751826] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1158.253135] Key type lgssc unregistered [ 1158.714052] LNet: 38606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.725477] LNetError: 38606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.754933] LNet: Removed LNI 192.168.204.154@tcp [ 1159.769438] Key type .llcrypt unregistered [ 1159.771619] Key type ._llcrypt unregistered [ 1184.804271] Key type ._llcrypt registered [ 1184.807288] Key type .llcrypt registered [ 1184.916211] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1186.202417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.276959] alg: No test for adler32 (adler32-zlib) [ 1187.385319] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1187.622170] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1189.280186] Key type lgssc registered [ 1190.384834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1197.415174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1197.447987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.907659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.822383] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1209.319735] LustreError: 40040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.353490] LustreError: 40040:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1212.719836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.171578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.125472] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1226.493993] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1231.758158] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1238.804076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1239.186284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.351586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1245.573567] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1252.724679] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1264.099841] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.106176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.121553] Lustre: Skipped 2 previous similar messages [ 1264.125641] Lustre: Skipped 1 previous similar message [ 1265.606558] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1265.622940] Lustre: Skipped 1 previous similar message [ 1268.918787] Lustre: server umount lustre-MDT0000 complete [ 1269.222641] LustreError: 40040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.244129] LustreError: 40040:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1270.687057] LustreError: 40035:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.100966] LustreError: 40020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857481 with bad export cookie 3229549295438554892 [ 1273.108113] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.337855] LustreError: 42531:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.339140] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.373387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1278.369174] LustreError: 40037:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.370537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1278.387548] LustreError: 40037:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1278.397533] Lustre: Skipped 1 previous similar message [ 1279.394206] Lustre: server umount lustre-MDT0001 complete [ 1295.581568] Lustre: server umount lustre-OST0000 complete [ 1315.276329] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1318.010364] Key type lgssc unregistered [ 1318.365906] LNet: 43906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.369836] LNetError: 43906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.390187] LNet: Removed LNI 192.168.204.154@tcp [ 1319.253590] Key type .llcrypt unregistered [ 1319.257600] Key type ._llcrypt unregistered [ 1346.349575] Key type ._llcrypt registered [ 1346.354542] Key type .llcrypt registered [ 1346.543069] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1348.085542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.117603] alg: No test for adler32 (adler32-zlib) [ 1349.370925] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1349.558321] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1351.216163] Key type lgssc registered [ 1352.589152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.173995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.193648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.615839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.889269] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1371.120625] LustreError: 45325:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.139199] LustreError: 45325:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1375.106191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.635072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.424528] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1387.915709] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1394.037847] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1401.335860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.757311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.918136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1408.934434] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1417.331848] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1430.460640] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1430.499625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.510761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.529464] Lustre: Skipped 1 previous similar message [ 1433.880317] Lustre: server umount lustre-OST0000 complete [ 1455.085642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.089285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.112718] Lustre: Skipped 1 previous similar message [ 1455.125302] Lustre: Skipped 2 previous similar messages [ 1459.566110] Lustre: server umount lustre-MDT0000 complete [ 1460.194325] LustreError: 45320:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.202713] LustreError: 45320:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1464.484477] LustreError: 45305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857672 with bad export cookie 17673435882605757998 [ 1464.486265] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.495372] LustreError: 45305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1470.876775] Lustre: server umount lustre-MDT0001 complete [ 1482.517032] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1485.565882] Key type lgssc unregistered [ 1485.835839] LNet: 49038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.841196] LNetError: 49038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.890558] LNet: Removed LNI 192.168.204.154@tcp [ 1488.039983] Key type .llcrypt unregistered [ 1488.043089] Key type ._llcrypt unregistered [ 1515.478459] Key type ._llcrypt registered [ 1515.483074] Key type .llcrypt registered [ 1515.602772] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1517.242272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1517.271192] alg: No test for adler32 (adler32-zlib) [ 1518.343888] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1518.645633] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1520.368272] Key type lgssc registered [ 1521.837449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1531.996652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1532.027288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.770797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.534355] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1543.139482] LustreError: 50458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.166185] LustreError: 50458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1548.276033] LustreError: 50459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1549.466265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.874435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1555.380869] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1562.993909] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1570.728480] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1579.363628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1579.791606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.137709] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1594.982604] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1603.394399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1611.104401] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1614.246347] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1627.563276] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2715h10m32s [ 1629.243630] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1631.713540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.728840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.762586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.718766] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1632.730133] Lustre: Skipped 2 previous similar messages [ 1633.747772] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1633.770321] Lustre: Skipped 3 previous similar messages [ 1637.817689] Lustre: server umount lustre-MDT0000 complete [ 1638.821581] LustreError: 50455:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.178103] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1644.953057] LustreError: 53377:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.983849] LustreError: 53377:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1651.134348] LustreError: 50455:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.169839] LustreError: 50455:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1651.988313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.246871] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.659211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.736396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1653.672450] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1658.452083] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1663.530356] Lustre: DEBUG MARKER: Wait for df (36752) ... [ 1665.580924] Lustre: DEBUG MARKER: done [ 1671.143499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1671.153273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.170497] Lustre: Skipped 2 previous similar messages [ 1671.177667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.188341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.203594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.511974] Lustre: server umount lustre-OST0000 complete [ 1684.454898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.470718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.491949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.491963] Lustre: Skipped 3 previous similar messages [ 1689.719148] Lustre: server umount lustre-MDT0000 complete [ 1691.640340] LustreError: 50453:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.673633] LustreError: 50453:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1694.670425] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857902 with bad export cookie 12305586011181440549 [ 1694.677141] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1694.686425] LustreError: 50439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1695.120664] Lustre: server umount lustre-MDT0001 complete [ 1709.664569] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1714.011168] Key type lgssc unregistered [ 1714.374081] LNet: 55409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.389946] LNetError: 55409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.419653] LNet: Removed LNI 192.168.204.154@tcp [ 1715.830300] Key type .llcrypt unregistered [ 1715.834531] Key type ._llcrypt unregistered [ 1762.373844] Key type ._llcrypt registered [ 1762.382279] Key type .llcrypt registered [ 1762.518613] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1764.173571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.212578] alg: No test for adler32 (adler32-zlib) [ 1765.241350] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1765.565420] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1767.304133] Key type lgssc registered [ 1768.990402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.148398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1778.221106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.609048] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1779.651112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1779.927860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.245288] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1794.395709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.538872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1794.612235] Lustre: 57618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1794.947869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.397948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1810.176429] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1817.207761] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1826.242909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.549729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1826.811313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.144595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1835.809864] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1845.897580] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.815681] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2715h14m20s [ 1855.630057] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1862.960319] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1865.996522] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1866.106958] LustreError: 57625:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1886.146149] Lustre: lustre-MDT0000: Client 1fd806d6-66f7-4ec4-a5e6-f19e03397514 (at 192.168.204.54@tcp) reconnecting [ 1906.168363] LustreError: 57625:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1906.249824] Lustre: lustre-MDT0000: Client 1fd806d6-66f7-4ec4-a5e6-f19e03397514 (at 192.168.204.54@tcp) reconnecting [ 1908.597974] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1911.523380] Lustre: DEBUG MARKER: done [ 1919.466717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.468540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.482028] LustreError: Skipped 1 previous similar message [ 1919.485847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.496837] Lustre: Skipped 1 previous similar message [ 1919.522444] Lustre: Skipped 1 previous similar message [ 1924.596851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.605823] Lustre: Skipped 1 previous similar message [ 1924.776215] Lustre: server umount lustre-OST0000 complete [ 1929.707313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.720184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.731978] Lustre: Skipped 1 previous similar message [ 1929.744153] Lustre: Skipped 1 previous similar message [ 1935.108651] Lustre: server umount lustre-MDT0000 complete [ 1940.187290] LustreError: 59484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858148 with bad export cookie 14028754732712245006 [ 1940.188596] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.197953] LustreError: 59484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1940.745228] Lustre: server umount lustre-MDT0001 complete [ 1955.164161] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1958.796344] Key type lgssc unregistered [ 1959.133450] LNet: 61959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.140184] LNetError: 61959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.158029] LNet: Removed LNI 192.168.204.154@tcp [ 1960.214923] Key type .llcrypt unregistered [ 1960.220681] Key type ._llcrypt unregistered [ 2010.803644] Key type ._llcrypt registered [ 2010.806970] Key type .llcrypt registered [ 2011.085721] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2012.968761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.993476] alg: No test for adler32 (adler32-zlib) [ 2014.053948] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2014.367262] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2016.104220] Key type lgssc registered [ 2017.890053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2022.513475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2022.580542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.188635] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2024.233277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2024.422396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.483973] Lustre: Failing over lustre-MDT0000 [ 2025.865559] Lustre: server umount lustre-MDT0000 complete [ 2036.990986] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:17:22 (1787858242) [ 2048.249714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.969453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.041783] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2065.410736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.467708] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2065.486726] Lustre: 64763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2065.726104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2071.288296] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2079.941810] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.052895] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.271108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.616194] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2094.844560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.378579] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2102.972285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2111.924817] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2118.996505] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2128.866903] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.883586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.886941] Lustre: Skipped 2 previous similar messages [ 2128.905752] Lustre: Skipped 2 previous similar messages [ 2133.988200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.003782] Lustre: Skipped 2 previous similar messages [ 2134.659308] Lustre: server umount lustre-MDT0000 complete [ 2139.303811] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858347 with bad export cookie 15083391732159811638 [ 2139.315090] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.316043] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2139.697374] Lustre: server umount lustre-MDT0001 complete [ 2143.468606] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2149.685698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2160.089167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.608177] Lustre: 63684:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858352/real 1787858352] req@00000000700b890c x1874705223035648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.633154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.771977] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd152f8dd7b731919 [ 2164.783460] Lustre: MGC192.168.204.154@tcp: Connection restored to 0@lo (at 0@lo) [ 2165.085633] LustreError: 66611:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.102869] LustreError: 66611:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2165.177248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.287451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2166.178119] Lustre: 63684:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858357/real 1787858357] req@000000000d838cb8 x1874705223035776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2169.259981] LustreError: 66612:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.275448] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2170.275167] Lustre: 63685:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858362/real 1787858362] req@000000008ae75b24 x1874705223036032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2170.338918] LustreError: 68536:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.348021] LustreError: 68536:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2170.356393] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2175.460575] LustreError: 68537:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.467132] Lustre: 63686:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858367/real 1787858367] req@00000000e8acafbb x1874705223036160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.485774] LustreError: 68537:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2180.594987] LustreError: 68535:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.629639] LustreError: 68535:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2181.344383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.088461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.267653] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2188.911397] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2199.427653] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2207.251939] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2217.954391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2217.957529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.965195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.000638] Lustre: Skipped 2 previous similar messages [ 2223.074578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.086597] Lustre: Skipped 3 previous similar messages [ 2228.213118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.221656] Lustre: Skipped 3 previous similar messages [ 2231.264245] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2231.646198] Lustre: server umount lustre-MDT0000 complete [ 2233.316819] LustreError: 68535:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.338299] LustreError: 68535:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2236.589322] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858444 with bad export cookie 15083391732159813913 [ 2236.594337] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2236.612375] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2236.936992] Lustre: server umount lustre-MDT0001 complete [ 2244.967877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2254.822924] Lustre: 63683:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858446/real 1787858446] req@00000000513f9542 x1874705223068416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2254.851321] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.309954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.026493] LustreError: 63682:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000087bacc5 x1874705223069312/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2261.636431] LustreError: 69272:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.783401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.928390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2263.786689] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.912187] Lustre: 63683:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858459/real 1787858459] req@00000000f0303b1f x1874705223069056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2266.944619] Lustre: 63683:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2268.201502] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2279.398596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.043175] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2285.562272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2294.523219] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2300.016567] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2310.644028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.646925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2310.653658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.401784] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2322.594704] Lustre: server umount lustre-MDT0000 complete [ 2325.993610] LustreError: 71332:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.013243] LustreError: 71332:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 2327.306513] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858535 with bad export cookie 15083391732159815292 [ 2327.313938] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.328697] LustreError: 64756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2328.138619] Lustre: server umount lustre-MDT0001 complete [ 2337.191528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2341.425384] Lustre: DEBUG MARKER: files compared the same [ 2354.420886] Lustre: server umount lustre-OST0000 complete [ 2371.836156] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2375.178440] Key type lgssc unregistered [ 2375.557343] LNet: 74513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2375.563586] LNetError: 74513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2375.582276] LNet: Removed LNI 192.168.204.154@tcp [ 2376.565209] Key type .llcrypt unregistered [ 2376.568468] Key type ._llcrypt unregistered [ 2401.561488] Key type ._llcrypt registered [ 2401.563813] Key type .llcrypt registered [ 2401.679610] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2402.734950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2402.747555] alg: No test for adler32 (adler32-zlib) [ 2404.019026] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2404.271967] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2405.976308] Key type lgssc registered [ 2407.335849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2416.169482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2416.193053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.782338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.310857] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 2427.363382] LustreError: 75929:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.383442] LustreError: 75929:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2432.482753] LustreError: 75930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.461940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.981452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.336932] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 2447.454936] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2453.947665] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2461.754676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2462.190903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.225671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2469.207613] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug +malloc all [ 2477.568908] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2488.806106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.816927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.825205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.826444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.829621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.846292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.248043] Lustre: server umount lustre-OST0000 complete [ 2500.094867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.096232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2500.106806] Lustre: Skipped 1 previous similar message [ 2500.110941] Lustre: Skipped 1 previous similar message [ 2505.193091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.201702] Lustre: Skipped 2 previous similar messages [ 2510.304982] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2510.311568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.349764] Lustre: Skipped 1 previous similar message [ 2510.522287] Lustre: server umount lustre-MDT0000 complete [ 2515.427794] LustreError: 75924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.459989] LustreError: 75924:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2515.837699] LustreError: 78422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858723 with bad export cookie 10521101360413817983 [ 2515.839731] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.864766] LustreError: 78422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2516.144248] Lustre: server umount lustre-MDT0001 complete [ 2529.497971] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2532.612336] Key type lgssc unregistered [ 2532.963712] LNet: 79466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.983902] LNetError: 79466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.020729] LNet: Removed LNI 192.168.204.154@tcp [ 2534.072543] Key type .llcrypt unregistered [ 2534.076370] Key type ._llcrypt unregistered [ 2562.610934] Key type ._llcrypt registered [ 2562.616022] Key type .llcrypt registered [ 2562.805126] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2564.493636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.601624] alg: No test for adler32 (adler32-zlib) [ 2565.717360] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2566.094802] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2567.888876] Key type lgssc registered [ 2569.322728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2578.779916] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2578.802186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.400563] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2599.520679] LustreError: 80876:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2626.208217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.548768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2640.324330] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2648.930663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.099382] Lustre: *** cfs_fail_loc=706, val=0*** [ 2649.681870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.757819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2654.577995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2654.700476] LustreError: 80902:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.726585] LustreError: 80902:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2659.812045] LustreError: 80901:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.863075] LustreError: 81960:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.886485] LustreError: 81960:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2664.700526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.253650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.043911] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2678.494994] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2685.431434] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2690.529345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.535392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.545118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.051609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.054837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.164714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2696.175058] Lustre: Skipped 1 previous similar message [ 2701.291153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2701.301871] Lustre: Skipped 1 previous similar message [ 2702.304109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2702.410436] Lustre: server umount lustre-OST0000 complete [ 2706.409472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2706.420563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2706.420650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2706.438548] Lustre: Skipped 1 previous similar message [ 2706.453327] Lustre: Skipped 1 previous similar message [ 2716.643237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.655108] Lustre: Skipped 4 previous similar messages [ 2720.224710] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2720.475824] Lustre: server umount lustre-MDT0000 complete [ 2721.770437] LustreError: 81962:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.795912] LustreError: 81962:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2724.639953] LustreError: 82702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858932 with bad export cookie 13640657314776438825 [ 2724.650519] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.662957] LustreError: 82702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2724.967813] Lustre: server umount lustre-MDT0001 complete [ 2736.559951] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2739.474801] Key type lgssc unregistered [ 2739.825606] LNet: 84497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2739.840338] LNetError: 84497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2739.854908] LNet: Removed LNI 192.168.204.154@tcp [ 2740.929993] Key type .llcrypt unregistered [ 2740.933127] Key type ._llcrypt unregistered [ 2774.255810] Key type ._llcrypt registered [ 2774.261268] Key type .llcrypt registered [ 2774.449564] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2775.915511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2775.948052] alg: No test for adler32 (adler32-zlib) [ 2777.020562] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2777.380281] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2779.080322] Key type lgssc registered [ 2780.566085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2790.904347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2790.937296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.000690] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2809.402001] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.926838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.948123] LustreError: 86604:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.978749] LustreError: 86604:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2816.092383] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2819.042879] LustreError: 86608:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.060449] LustreError: 86608:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2824.162535] LustreError: 86607:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.206417] LustreError: 86607:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2827.490084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.010551] LustreError: 86608:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.058315] LustreError: 86608:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2828.064560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.378475] LustreError: 86607:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.414175] LustreError: 86607:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2835.009572] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2844.154696] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2844.181126] Lustre: MGS: Client 74f56a06-6d14-4c51-a7d8-5b43162d3fc5 (at 0@lo) reconnecting [ 2844.200334] LustreError: 86618:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2844.436275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.514630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2849.358294] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2865.126328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.130955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.137584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.141679] Lustre: Skipped 1 previous similar message [ 2865.157193] Lustre: Skipped 1 previous similar message [ 2870.233370] Lustre: server umount lustre-OST0000 complete [ 2875.364059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.364185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2875.375725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.393165] Lustre: Skipped 1 previous similar message [ 2880.484522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.505481] Lustre: Skipped 1 previous similar message [ 2885.605680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.626294] Lustre: Skipped 1 previous similar message [ 2888.160280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2888.261780] Lustre: server umount lustre-MDT0000 complete [ 2890.722498] LustreError: 88524:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.740018] LustreError: 88524:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 2893.914920] Lustre: server umount lustre-MDT0001 complete [ 2898.498653] Lustre: server umount MGS complete [ 2911.877861] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:31:57 (1787859117) [ 2945.788834] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2957.327780] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2996.212436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.252465] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3009.306990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.482441] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3009.518399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3009.592821] Lustre: lustre-MDT0001: new disk, initializing [ 3009.683100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.700387] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3009.708966] Lustre: 93795:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3009.719897] LustreError: 93795:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3013.860310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3025.028663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.404744] Lustre: lustre-OST0000: new disk, initializing [ 3025.411860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3025.424848] Lustre: 94562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3025.560138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.099587] Lustre: 94853:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000087bacc5 x1874706023097856/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787859256 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3032.325597] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3042.843659] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3050.999732] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.033719] Lustre: MGS: Client 6f38a039-f2f9-4a4e-8c77-a8bfd31f8a07 (at 0@lo) reconnecting [ 3051.062511] LustreError: 93781:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3051.100472] Lustre: 93063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3051.183864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3051.334397] Lustre: lustre-MDT0000: new disk, initializing [ 3051.507419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.533497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3051.571374] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3052.649931] Lustre: 95631:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a9e4c589 x1874706023104896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787859276 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3056.692866] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3057.770143] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3061.809054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3061.822123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3061.873059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3091.442448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.450725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3091.460810] Lustre: Skipped 2 previous similar messages [ 3091.480317] Lustre: Skipped 1 previous similar message [ 3095.597781] Lustre: server umount lustre-OST0000 complete [ 3101.668297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3101.669552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.684869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.727346] Lustre: Skipped 1 previous similar message [ 3106.075441] Lustre: server umount lustre-MDT0000 complete [ 3106.787286] LustreError: 93767:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.816360] LustreError: 93767:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3116.614748] Lustre: server umount lustre-MDT0001 complete [ 3121.087151] Lustre: server umount MGS complete [ 3133.185800] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:35:39 (1787859339) [ 3151.389149] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3154.532187] Key type lgssc unregistered [ 3154.883751] LNet: 98136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3154.890391] LNetError: 98136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3154.907080] LNet: Removed LNI 192.168.204.154@tcp [ 3156.097876] Key type .llcrypt unregistered [ 3156.101458] Key type ._llcrypt unregistered [ 3173.798721] Key type ._llcrypt registered [ 3173.801520] Key type .llcrypt registered [ 3173.903407] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3175.061344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.076083] alg: No test for adler32 (adler32-zlib) [ 3176.102697] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 3176.357496] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3178.072666] Key type lgssc registered [ 3179.203553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.116862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3186.132673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3187.509367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.725677] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3196.911487] LustreError: 99359:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.925433] LustreError: 99359:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3200.002614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.398868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3205.407046] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3214.097869] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3220.900914] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3229.165047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3229.452189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.570016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3236.272949] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3244.530794] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3253.222221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3253.233299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.246926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.297983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3256.301421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.315372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.691607] Lustre: server umount lustre-OST0000 complete [ 3266.538067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.540625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.554268] Lustre: Skipped 1 previous similar message [ 3266.573597] Lustre: Skipped 2 previous similar messages [ 3269.097292] Lustre: server umount lustre-MDT0000 complete [ 3271.650684] LustreError: 100983:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.669749] LustreError: 100983:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3272.844087] LustreError: 99338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859480 with bad export cookie 10220872557254848197 [ 3272.848498] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.857160] LustreError: 99338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3273.195688] Lustre: server umount lustre-MDT0001 complete [ 3287.084932] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3290.590751] Key type lgssc unregistered [ 3291.054663] LNet: 102894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.072498] LNetError: 102894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.100636] LNet: Removed LNI 192.168.204.154@tcp [ 3292.125426] Key type .llcrypt unregistered [ 3292.127756] Key type ._llcrypt unregistered [ 3309.669864] Key type ._llcrypt registered [ 3309.681140] Key type .llcrypt registered [ 3309.812588] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3311.185328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3311.245738] alg: No test for adler32 (adler32-zlib) [ 3312.338334] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 3312.628690] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3314.297068] Key type lgssc registered [ 3315.735563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3322.050552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3322.054321] LustreError: 103977:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3322.067361] LustreError: 103980:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3322.072549] LustreError: 103977:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3322.081223] LustreError: 103977:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3322.093111] LustreError: 103977:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3322.096568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.669763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.008582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.181820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3334.320973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3340.101390] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing lsmod [ 3343.234299] LustreError: 105184:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3343.235783] LustreError: 105185:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3343.254655] LustreError: 105184:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3343.260808] LustreError: 105184:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3343.275611] LustreError: 105184:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3343.291550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.746074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.323669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.336482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3346.347406] Lustre: Skipped 1 previous similar message [ 3346.363896] Lustre: Skipped 2 previous similar messages [ 3351.525500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.535820] Lustre: Skipped 2 previous similar messages [ 3352.755304] Lustre: server umount lustre-MDT0000 complete [ 3355.621498] LustreError: 104019:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.668267] LustreError: 104019:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 3356.782669] LustreError: 103993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859564 with bad export cookie 14210491900398200994 [ 3356.785355] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.804271] LustreError: 103993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.793315] LustreError: 104449:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d1e2b579 x1874706584063360/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3357.961847] Lustre: server umount lustre-MDT0001 complete [ 3372.220427] Lustre: server umount lustre-OST0000 complete [ 3381.867880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.311952] LustreError: 106249:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.377477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.368279] LustreError: 106250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.414341] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3392.488488] LustreError: 106249:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.611888] LustreError: 106250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.590806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.403780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3405.391212] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3414.864398] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3420.642417] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3428.216453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3428.341081] LustreError: 108076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.703101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.971136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3434.486221] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3442.003639] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3459.566539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.566701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.588087] Lustre: Skipped 1 previous similar message [ 3459.600731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3461.597992] Lustre: server umount lustre-OST0000 complete [ 3469.805681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.836711] Lustre: Skipped 1 previous similar message [ 3469.861955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.885246] Lustre: Skipped 2 previous similar messages [ 3473.536827] Lustre: server umount lustre-MDT0000 complete [ 3474.915885] LustreError: 106250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.938140] LustreError: 106250:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3478.472117] LustreError: 106229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859686 with bad export cookie 14210491900398201687 [ 3478.487527] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.495200] LustreError: 106229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3478.772269] Lustre: server umount lustre-MDT0001 complete [ 3491.441948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3495.355958] Key type lgssc unregistered [ 3495.592409] LNet: 109811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3495.605916] LNetError: 109811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3495.626729] LNet: Removed LNI 192.168.204.154@tcp [ 3496.576138] Key type .llcrypt unregistered [ 3496.583276] Key type ._llcrypt unregistered [ 3523.988780] Key type ._llcrypt registered [ 3523.995866] Key type .llcrypt registered [ 3524.128584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3525.514128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3525.547280] alg: No test for adler32 (adler32-zlib) [ 3526.668892] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 3526.975799] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3528.688169] Key type lgssc registered [ 3530.071734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3538.793494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3538.821110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.388890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.117360] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3550.692504] LustreError: 111243:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.715165] LustreError: 111243:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3553.811417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.396947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3560.147477] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3568.629383] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3575.051844] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3584.396067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.777784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.871841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3591.018897] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3600.486782] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3605.487717] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3609.010324] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3614.692943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.692132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3614.698699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.702662] Lustre: Skipped 1 previous similar message [ 3619.108484] Lustre: server umount lustre-OST0000 complete [ 3624.931425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.933127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.942910] Lustre: Skipped 1 previous similar message [ 3624.957034] Lustre: Skipped 2 previous similar messages [ 3630.050103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.055451] Lustre: Skipped 1 previous similar message [ 3634.150754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.160316] Lustre: Skipped 1 previous similar message [ 3637.216251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3637.446696] Lustre: server umount lustre-MDT0000 complete [ 3639.270253] LustreError: 113924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.293528] LustreError: 113924:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3641.551873] LustreError: 111224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859849 with bad export cookie 3301282951746368684 [ 3641.566026] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.579652] LustreError: 111224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3642.123446] Lustre: server umount lustre-MDT0001 complete [ 3655.778276] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3659.137440] Key type lgssc unregistered [ 3659.449394] LNet: 114882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3659.459359] LNetError: 114882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3660.538903] LNet: Removed LNI 192.168.204.154@tcp [ 3661.589259] Key type .llcrypt unregistered [ 3661.593337] Key type ._llcrypt unregistered [ 3681.434504] Key type ._llcrypt registered [ 3681.440970] Key type .llcrypt registered [ 3681.513450] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3682.391969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3682.453211] alg: No test for adler32 (adler32-zlib) [ 3683.619376] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 3683.878039] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3685.586777] Key type lgssc registered [ 3686.670087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3696.314393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3696.339904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.994720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.445974] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3708.390841] LustreError: 116107:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.439851] LustreError: 116107:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3713.509462] LustreError: 116108:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.875779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.424893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3720.212799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3728.050671] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3735.342383] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3742.562498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3742.891246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.932503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3749.299606] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3756.758112] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3769.850954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3769.851327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.882692] Lustre: Skipped 1 previous similar message [ 3769.897302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.908639] Lustre: Skipped 1 previous similar message [ 3772.708389] Lustre: server umount lustre-OST0000 complete [ 3780.067242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.070474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.084865] Lustre: Skipped 1 previous similar message [ 3780.095936] Lustre: Skipped 1 previous similar message [ 3784.488590] Lustre: server umount lustre-MDT0000 complete [ 3785.190853] LustreError: 116107:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.215891] LustreError: 116107:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3788.728533] LustreError: 116088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859996 with bad export cookie 958640384817197791 [ 3788.745913] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3788.999763] Lustre: server umount lustre-MDT0001 complete [ 3803.264930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3806.278824] Key type lgssc unregistered [ 3806.684525] LNet: 119645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3806.698396] LNetError: 119645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3806.725682] LNet: Removed LNI 192.168.204.154@tcp [ 3807.766388] Key type .llcrypt unregistered [ 3807.778415] Key type ._llcrypt unregistered [ 3834.985023] Key type ._llcrypt registered [ 3834.986841] Key type .llcrypt registered [ 3835.136094] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3836.482391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3836.514873] alg: No test for adler32 (adler32-zlib) [ 3837.547024] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 3837.728427] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3839.448183] Key type lgssc registered [ 3841.054773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3848.753385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3848.770095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.140367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.588377] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3860.451037] LustreError: 121066:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.466824] LustreError: 121066:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3865.579431] LustreError: 121065:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.763244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.154490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.635339] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3878.793500] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3885.185520] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3893.505436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3893.918907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.975737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3901.934669] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3910.375699] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3918.677976] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3923.309314] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3923.314356] Lustre: Skipped 2 previous similar messages [ 3926.125830] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3926.191804] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3928.480479] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3928.487809] Lustre: Skipped 2 previous similar messages [ 3935.826784] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3935.836085] Lustre: Skipped 1 previous similar message [ 3941.977869] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3941.985226] Lustre: Skipped 2 previous similar messages [ 3943.543658] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.54@tcp 192.168.0.[2,10]@tcp [ 3956.306872] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.54/24@tcp in log lustre-MDT0000 [ 3956.326688] Lustre: Skipped 2 previous similar messages [ 3963.488416] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.54/24@tcp [ 3963.498319] Lustre: Skipped 1 previous similar message [ 3972.089081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3972.116744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.152684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.090604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.093896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.101931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.129120] Lustre: Skipped 1 previous similar message [ 3976.849581] Lustre: server umount lustre-OST0000 complete [ 3983.335041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.338597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.351302] Lustre: Skipped 1 previous similar message [ 3983.361927] Lustre: Skipped 1 previous similar message [ 3987.408468] Lustre: server umount lustre-MDT0000 complete [ 3988.465897] LustreError: 121066:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.489278] LustreError: 121066:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3992.449229] LustreError: 121045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860200 with bad export cookie 14023459394304758466 [ 3992.451109] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.463350] LustreError: 121045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3993.220926] Lustre: server umount lustre-MDT0001 complete [ 4006.307710] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4009.966496] Key type lgssc unregistered [ 4010.353835] LNet: 126017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4010.358516] LNetError: 126017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4010.399207] LNet: Removed LNI 192.168.204.154@tcp [ 4011.699814] Key type .llcrypt unregistered [ 4011.703483] Key type ._llcrypt unregistered [ 4043.453400] Key type ._llcrypt registered [ 4043.456507] Key type .llcrypt registered [ 4043.549746] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4044.638653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4044.662390] alg: No test for adler32 (adler32-zlib) [ 4045.773200] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 4046.133429] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4047.872189] Key type lgssc registered [ 4048.922880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4056.506662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4056.518565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.916438] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4057.920915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.940965] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.54/24@tcp [ 4062.983254] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4068.333239] LustreError: 127597:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.358137] LustreError: 127597:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4071.473242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.841500] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4071.860864] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.54/24@tcp [ 4071.942594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4076.360726] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4084.200599] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4090.632665] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4098.452618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4098.916900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.034089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4106.255932] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4114.610808] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4126.697261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.707155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4126.720561] Lustre: Skipped 1 previous similar message [ 4126.722591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4126.747530] Lustre: Skipped 1 previous similar message [ 4131.280846] Lustre: server umount lustre-OST0000 complete [ 4136.931677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.947685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.958492] Lustre: Skipped 1 previous similar message [ 4142.056164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4142.069032] Lustre: Skipped 1 previous similar message [ 4142.717972] Lustre: server umount lustre-MDT0000 complete [ 4146.146178] LustreError: 127596:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.177988] LustreError: 127596:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4148.015845] LustreError: 130230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860356 with bad export cookie 4774727480878895178 [ 4148.018098] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.023969] LustreError: 130230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4148.518433] Lustre: server umount lustre-MDT0001 complete [ 4161.274152] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4164.925603] Key type lgssc unregistered [ 4165.356559] LNet: 131189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4165.367087] LNetError: 131189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.387795] LNet: Removed LNI 192.168.204.154@tcp [ 4166.482574] Key type .llcrypt unregistered [ 4166.487405] Key type ._llcrypt unregistered [ 4200.055628] Key type ._llcrypt registered [ 4200.058661] Key type .llcrypt registered [ 4200.188971] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4215.444463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4216.475107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4216.527550] alg: No test for adler32 (adler32-zlib) [ 4217.532785] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 4217.760750] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4219.512445] Key type lgssc registered [ 4221.596698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4263.848145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4263.874292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.179279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4265.228219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4265.320242] Lustre: lustre-MDT0000: new disk, initializing [ 4265.399253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.418296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4271.020784] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4283.582039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.735809] Lustre: 134590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4283.804577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4283.808245] Lustre: Skipped 1 previous similar message [ 4283.881275] Lustre: lustre-MDT0001: new disk, initializing [ 4283.965885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4283.995939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.008428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4289.855838] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4300.441758] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4307.109712] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4314.280623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.620523] Lustre: lustre-OST0000: new disk, initializing [ 4314.626623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4314.640959] Lustre: 136562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4314.782021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.282508] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4321.860590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4321.884178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4322.017650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4331.629457] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4337.311779] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4337.580395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4341.661348] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4341.871667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4347.879261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4347.881496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.902083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4347.922986] Lustre: Skipped 1 previous similar message [ 4347.936880] Lustre: Skipped 1 previous similar message [ 4350.360292] Lustre: server umount lustre-OST0000 complete [ 4355.555618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4355.569181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.578183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.122318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.123552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.803967] Lustre: server umount lustre-MDT0000 complete [ 4365.476072] LustreError: 134584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860573 with bad export cookie 8456366620024998246 [ 4365.481835] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.490828] LustreError: 134584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4365.906924] Lustre: server umount lustre-MDT0001 complete [ 4377.306940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.068945] LustreError: 138776:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.110728] LustreError: 138776:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4378.178881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.838204] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4383.206838] LustreError: 138777:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.324377] LustreError: 138776:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.383876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.631200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4397.196799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4405.543628] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4412.510970] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4420.549463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4420.870394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.292833] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4437.686494] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4443.274557] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4443.508808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4449.377572] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4449.728059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4461.095572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4461.304320] Lustre: lustre-OST0001: new disk, initializing [ 4461.313581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4461.329459] Lustre: 142151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4461.453037] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4468.121409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4470.902591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4470.925830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4471.043683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4477.830803] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4478.111193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4483.011467] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4483.270413] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4489.470564] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c9403744000.ost_server_uuid 50 [ 4491.810079] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c9403744000.ost_server_uuid in IDLE state after 0 sec [ 4502.010069] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.017195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4502.026125] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4502.026135] Lustre: Skipped 1 previous similar message [ 4502.030433] Lustre: Skipped 1 previous similar message [ 4502.061115] LustreError: Skipped 1 previous similar message [ 4507.119327] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4507.131347] Lustre: Skipped 2 previous similar messages [ 4516.320147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4516.436282] Lustre: server umount lustre-OST0001 complete [ 4517.355931] LustreError: 141428:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.374995] LustreError: 141428:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4522.469229] LustreError: 140599:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.475312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.483851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4522.483860] Lustre: Skipped 2 previous similar messages [ 4522.504151] LustreError: 140599:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4522.522283] Lustre: Skipped 1 previous similar message [ 4526.926729] Lustre: server umount lustre-OST0000 complete [ 4532.706177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.795996] Lustre: server umount lustre-MDT0000 complete [ 4537.826400] LustreError: 138777:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.846186] LustreError: 138777:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4542.431769] LustreError: 138759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860750 with bad export cookie 8456366620024999128 [ 4542.441507] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.447312] LustreError: 138759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4542.942092] Lustre: server umount lustre-MDT0001 complete [ 4566.126887] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4569.167092] Key type lgssc unregistered [ 4569.532961] LNet: 145158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4569.549333] LNetError: 145158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4569.570296] LNet: Removed LNI 192.168.204.154@tcp [ 4570.633776] Key type .llcrypt unregistered [ 4570.643550] Key type ._llcrypt unregistered [ 4615.823043] Key type ._llcrypt registered [ 4615.828489] Key type .llcrypt registered [ 4615.977503] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4636.406403] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4638.024220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4638.136922] alg: No test for adler32 (adler32-zlib) [ 4639.228049] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 4639.499029] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4641.225967] Key type lgssc registered [ 4642.449250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4683.033359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4683.052347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.372712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4684.414678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4684.475501] Lustre: lustre-MDT0000: new disk, initializing [ 4684.630680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.669389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4690.190607] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4702.246319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.411153] Lustre: 149328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4702.479035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4702.483944] Lustre: Skipped 1 previous similar message [ 4702.573964] Lustre: lustre-MDT0001: new disk, initializing [ 4702.715215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4702.760445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4702.783140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4709.051379] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4721.269558] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4728.861933] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4737.756097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4738.242656] Lustre: lustre-OST0000: new disk, initializing [ 4738.249492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4738.256754] Lustre: 151300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4738.414826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.210596] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4746.835228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4746.850570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4747.004350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4756.621121] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4760.927995] Lustre: Failing over lustre-OST0000 [ 4761.236116] Lustre: server umount lustre-OST0000 complete [ 4777.952167] Lustre: 147545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860970/real 1787860970] req@0000000007fb2994 x1874707975539712/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787860986 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4777.981454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.047635] Lustre: 147544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860970/real 1787860970] req@000000008942dd87 x1874707975538688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787860986 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4779.083885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4780.798648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4781.303746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.327941] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4783.330332] Lustre: 147544:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860975/real 1787860975] req@00000000fc136a8a x1874707975540864/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787860991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4783.391462] Lustre: 147544:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4787.931692] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4788.448743] Lustre: 147542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860980/real 1787860980] req@00000000650132b7 x1874707975542016/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787860996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4788.460404] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4788.480036] Lustre: 147542:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4789.747345] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4789.749674] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4791.947089] Lustre: Failing over lustre-MDT0000 [ 4792.249674] Lustre: server umount lustre-MDT0000 complete [ 4793.574070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.596729] Lustre: Skipped 2 previous similar messages [ 4794.593199] Lustre: 147545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787860986/real 1787860986] req@0000000033e369e4 x1874707975543936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787861002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4794.624412] Lustre: 147545:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4798.947300] LustreError: 149342:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4798.977493] LustreError: 149342:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 4799.896149] LustreError: 149335:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.069743] LustreError: 153215:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.105922] LustreError: 153215:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4809.184776] Lustre: 147545:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861001/real 1787861001] req@00000000e2fd440f x1874707975555456/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787861017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4809.193531] LustreError: 153215:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.217316] Lustre: 147545:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4809.223898] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4809.231021] LustreError: 153215:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4811.813722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.309617] LustreError: 151880:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.332107] LustreError: 151880:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4819.427587] LustreError: 147541:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa867803 x1874707975560320/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4819.814896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.899640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4820.259131] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4824.694098] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4825.087655] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4825.096203] Lustre: Skipped 1 previous similar message [ 4825.135315] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4825.191369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4835.301429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.310409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4835.321329] Lustre: Skipped 1 previous similar message [ 4835.331495] Lustre: Skipped 1 previous similar message [ 4840.323133] Lustre: server umount lustre-OST0000 complete [ 4845.538806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.547529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4845.553961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4850.642563] Lustre: server umount lustre-MDT0000 complete [ 4850.664518] LustreError: 153814:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.705474] LustreError: 153814:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 4855.160105] LustreError: 149323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861063 with bad export cookie 2352545698799396881 [ 4855.172498] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.184864] LustreError: 149323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4855.842189] Lustre: server umount lustre-MDT0001 complete [ 4868.194876] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4872.167093] Key type lgssc unregistered [ 4872.550487] LNet: 154961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4872.559550] LNetError: 154961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4872.601869] LNet: Removed LNI 192.168.204.154@tcp [ 4873.743677] Key type .llcrypt unregistered [ 4873.746846] Key type ._llcrypt unregistered [ 4913.776519] Key type ._llcrypt registered [ 4913.784405] Key type .llcrypt registered [ 4913.906526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4915.435894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4915.483376] alg: No test for adler32 (adler32-zlib) [ 4916.692345] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 4916.978782] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4918.736532] Key type lgssc registered [ 4920.317980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4928.940857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4928.951847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.376094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.920624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4940.769308] LustreError: 156484:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.788820] LustreError: 156484:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4943.767332] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4945.893659] LustreError: 156483:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.358639] LustreError: 156512:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4947.375722] LustreError: 156512:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4947.386807] LustreError: 156512:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4947.738630] Lustre: server umount lustre-MDT0000 complete [ 4962.240796] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4970.884228] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5009.364472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.600670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5009.639577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5009.707326] Lustre: lustre-MDT0000: new disk, initializing [ 5009.787615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.802121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5014.211904] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5026.257938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.337566] Lustre: 159816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5026.404911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5026.408395] Lustre: Skipped 1 previous similar message [ 5026.513615] Lustre: lustre-MDT0001: new disk, initializing [ 5026.696045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5026.738713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5026.757830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5032.552882] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5042.692608] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5048.978931] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5056.454683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.806889] Lustre: lustre-OST0000: new disk, initializing [ 5056.814608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5056.821539] Lustre: 161783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5056.940663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.997423] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5065.418911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5065.440437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5065.567494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5074.208804] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5078.324440] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5080.374304] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:08:06 (1787861286) [ 5374.944810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5374.956822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.978056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.019098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.026372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.029287] Lustre: Skipped 1 previous similar message [ 5378.042203] Lustre: Skipped 1 previous similar message [ 5383.137275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.143242] Lustre: Skipped 2 previous similar messages [ 5388.257213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5388.266092] Lustre: Skipped 2 previous similar messages [ 5389.280461] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5391.509927] Lustre: server umount lustre-MDT0000 complete [ 5393.385083] LustreError: 163536:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.404115] LustreError: 163536:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5395.442812] LustreError: 159809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861603 with bad export cookie 16171893686119484587 [ 5395.449644] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.460443] LustreError: 159809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5398.496894] LustreError: 159823:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.498332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.546107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5401.862964] Lustre: server umount lustre-MDT0001 complete [ 5412.152340] Lustre: server umount lustre-OST0000 complete [ 5423.480200] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5435.615309] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5476.723169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.030265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5477.102502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5477.247521] Lustre: lustre-MDT0000: new disk, initializing [ 5477.332836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.344192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5481.729622] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5492.381044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.469425] Lustre: 166457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5492.528167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5492.534137] Lustre: Skipped 1 previous similar message [ 5492.603610] Lustre: lustre-MDT0001: new disk, initializing [ 5492.703862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5492.740197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5492.756836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5497.872794] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5507.403655] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5513.035559] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5520.977734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5521.207225] Lustre: lustre-OST0000: new disk, initializing [ 5521.211199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5521.217391] Lustre: 168425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5521.322959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.465715] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5531.189040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5531.199251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5531.354274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5539.041834] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5546.074576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5546.412091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5552.345810] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5552.483228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5557.222486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5557.228245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.243861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5557.261131] Lustre: Skipped 1 previous similar message [ 5557.270575] Lustre: Skipped 1 previous similar message [ 5561.426801] Lustre: server umount lustre-OST0000 complete [ 5567.457733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.475047] Lustre: Skipped 1 previous similar message [ 5572.999922] Lustre: server umount lustre-MDT0000 complete [ 5577.073574] LustreError: 166451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861785 with bad export cookie 16171893686119643417 [ 5577.093291] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5577.093568] LustreError: 166451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5577.635766] Lustre: server umount lustre-MDT0001 complete [ 5587.307225] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:16:33 (1787861793) [ 5605.898894] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5617.859674] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5658.940773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.370791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5659.421198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5659.504570] Lustre: lustre-MDT0000: new disk, initializing [ 5659.618471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.646447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5665.595308] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5674.945412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.035327] Lustre: 173185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5675.309453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5679.909319] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5690.890244] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5697.657858] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.307873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.583413] Lustre: lustre-OST0000: new disk, initializing [ 5705.586345] Lustre: Skipped 1 previous similar message [ 5705.589630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5705.597922] Lustre: Skipped 2 previous similar messages [ 5705.604272] Lustre: 175163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5710.957176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5710.965955] Lustre: Skipped 1 previous similar message [ 5710.969235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5711.122158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5711.889472] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5722.212851] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5731.814258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.815262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5731.841979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.860783] LustreError: Skipped 1 previous similar message [ 5731.878631] Lustre: Skipped 5 previous similar messages [ 5736.533045] Lustre: server umount lustre-OST0000 complete [ 5742.050225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5742.058317] LustreError: Skipped 1 previous similar message [ 5754.336529] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5754.588751] Lustre: server umount lustre-MDT0000 complete [ 5757.410587] LustreError: 173196:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.448360] LustreError: 173196:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5760.330125] LustreError: 173984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861968 with bad export cookie 16171893686119644299 [ 5760.337182] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.357492] LustreError: 173984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5760.758720] Lustre: server umount lustre-MDT0001 complete [ 5773.734672] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:19:39 (1787861979) [ 5789.578364] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5800.647610] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5843.280832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.676374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5843.709381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5843.781728] Lustre: lustre-MDT0000: new disk, initializing [ 5843.915523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.934124] Lustre: Skipped 2 previous similar messages [ 5843.961204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5849.283464] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5861.182210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.324672] Lustre: 179487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5861.715189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5867.583936] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5879.650375] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5889.686605] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5900.010533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5900.532330] Lustre: 181466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5907.753421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5908.048515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5908.235951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5919.151744] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5928.936742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.937040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.948930] LustreError: Skipped 1 previous similar message [ 5928.953943] Lustre: Skipped 3 previous similar messages [ 5928.956236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.961783] Lustre: Skipped 7 previous similar messages [ 5935.049388] Lustre: server umount lustre-OST0000 complete [ 5943.780502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5945.762608] Lustre: server umount lustre-MDT0000 complete [ 5949.412527] LustreError: 179500:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.462533] LustreError: 179500:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5950.787858] LustreError: 179480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862158 with bad export cookie 16171893686119645384 [ 5950.789323] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5950.803688] LustreError: 179480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5951.155778] Lustre: server umount lustre-MDT0001 complete [ 5966.953270] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5970.537919] Key type lgssc unregistered [ 5970.911458] LNet: 183325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5970.921147] LNetError: 183325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5970.945675] LNet: Removed LNI 192.168.204.154@tcp [ 5971.923872] Key type .llcrypt unregistered [ 5971.925437] Key type ._llcrypt unregistered [ 6000.432281] Key type ._llcrypt registered [ 6000.435805] Key type .llcrypt registered [ 6000.527915] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6001.518132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6001.696895] alg: No test for adler32 (adler32-zlib) [ 6002.806730] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 6003.021233] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6004.680182] Key type lgssc registered [ 6005.922386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6014.736406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6014.767080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6016.389924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.153931] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6026.728451] LustreError: 184744:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.749502] LustreError: 184744:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6031.566791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.843115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6031.986306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6037.110608] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6046.130806] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6054.094502] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6061.991458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.350206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.725720] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6079.597848] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6085.943182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6108.641290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6108.656987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.676397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.719060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.721436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6111.749913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.633520] Lustre: server umount lustre-OST0000 complete [ 6121.960889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.981801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.988942] Lustre: Skipped 2 previous similar messages [ 6126.070618] Lustre: server umount lustre-MDT0000 complete [ 6127.081727] LustreError: 184739:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6127.113737] LustreError: 184739:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6130.778176] LustreError: 187549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862338 with bad export cookie 14088225444210301391 [ 6130.797026] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6131.263273] Lustre: server umount lustre-MDT0001 complete [ 6145.910363] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6150.009403] Key type lgssc unregistered [ 6150.388787] LNet: 188304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6150.398651] LNetError: 188304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6150.432318] LNet: Removed LNI 192.168.204.154@tcp [ 6151.685265] Key type .llcrypt unregistered [ 6151.697320] Key type ._llcrypt unregistered [ 6182.916786] Key type ._llcrypt registered [ 6182.919594] Key type .llcrypt registered [ 6183.066365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6184.222930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6184.247973] alg: No test for adler32 (adler32-zlib) [ 6185.352609] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 6185.756699] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6187.576323] Key type lgssc registered [ 6189.211232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.110580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6199.128361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.873128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.299675] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6210.030115] LustreError: 189726:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.070469] LustreError: 189726:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6215.149359] LustreError: 189727:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.030764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.703905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6222.351147] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6232.021040] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6239.618591] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6248.405286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6248.918190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.489986] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6265.780322] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6271.490175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6278.069285] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6281.697092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6281.703025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.718052] LustreError: Skipped 1 previous similar message [ 6281.728307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6281.740558] Lustre: Skipped 1 previous similar message [ 6281.759765] Lustre: Skipped 1 previous similar message [ 6283.164086] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6283.931047] Lustre: server umount lustre-OST0000 complete [ 6292.586256] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8c9409c89000.ost_server_uuid 50 [ 6294.572771] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6311.397984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.400403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.410096] Lustre: Skipped 1 previous similar message [ 6311.430020] Lustre: Skipped 1 previous similar message [ 6315.918875] Lustre: server umount lustre-MDT0000 complete [ 6316.520648] LustreError: 190454:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.551856] LustreError: 190454:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6320.307525] LustreError: 189707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862528 with bad export cookie 17366323752605190534 [ 6320.343772] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6320.861152] Lustre: server umount lustre-MDT0001 complete [ 6331.630788] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:28:57 (1787862537) [ 6340.143340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.803572] LustreError: 193648:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.881332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6346.211489] LustreError: 193649:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6346.771843] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6355.439587] LustreError: 193648:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.479056] LustreError: 193648:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6356.159919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.597266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.948439] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6370.683451] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6377.033826] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6385.270553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.737620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.476278] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6399.634894] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.446208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6407.666292] Lustre: lustre-OST0001: new disk, initializing [ 6407.677640] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6407.684508] Lustre: 196490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6407.823549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6410.874688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6413.391513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6413.399433] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6413.510188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6414.393508] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6424.986489] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6429.161131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.167459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6429.179830] Lustre: Skipped 1 previous similar message [ 6429.191927] Lustre: Skipped 1 previous similar message [ 6434.227955] Lustre: server umount lustre-OST0000 complete [ 6434.276679] LustreError: 195472:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.324912] LustreError: 195472:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6442.392810] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6442.737643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6447.679936] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6447.946188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6464.997300] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.998640] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6465.006188] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6465.006200] Lustre: Skipped 1 previous similar message [ 6465.020378] Lustre: Skipped 1 previous similar message [ 6465.030947] LustreError: Skipped 1 previous similar message [ 6468.218860] Lustre: server umount lustre-OST0001 complete [ 6474.213442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6474.234732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6479.179654] Lustre: server umount lustre-MDT0000 complete [ 6480.355053] LustreError: 193644:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.382295] LustreError: 193644:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 18 previous similar messages [ 6483.832387] LustreError: 193629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862691 with bad export cookie 17366323752605191612 [ 6483.834061] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.841833] LustreError: 193629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6484.195467] Lustre: server umount lustre-MDT0001 complete [ 6509.545368] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:31:55 (1787862715) [ 6517.769731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6518.370935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6518.417847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6518.630384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.940613] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6533.556694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.764551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6533.853700] Lustre: 199934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6539.615319] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6548.997386] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6555.329399] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6563.290682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.658732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6571.586566] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6574.275243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6580.645831] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6587.624061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.717502] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6588.057486] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6588.067645] Lustre: Skipped 2 previous similar messages [ 6593.798417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6595.501856] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6604.758356] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.382628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.389898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.394727] Lustre: Skipped 2 previous similar messages [ 6609.403157] Lustre: Skipped 5 previous similar messages [ 6613.780369] Lustre: server umount lustre-OST0000 complete [ 6614.506240] LustreError: 201784:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6614.531165] LustreError: 201784:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6634.997412] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6634.998132] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6635.027083] Lustre: Skipped 1 previous similar message [ 6636.920190] Lustre: server umount lustre-OST0001 complete [ 6641.633221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6641.642335] LustreError: Skipped 1 previous similar message [ 6641.653841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6641.663353] Lustre: Skipped 3 previous similar messages [ 6647.646841] Lustre: server umount lustre-MDT0000 complete [ 6652.068708] LustreError: 202820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862860 with bad export cookie 17366323752605193019 [ 6652.076892] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.093906] LustreError: 202820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6652.533055] Lustre: server umount lustre-MDT0001 complete [ 6683.657524] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:34:49 (1787862889) [ 6704.677269] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 6716.664047] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6756.983059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.210739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6757.263937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6757.341808] Lustre: lustre-MDT0000: new disk, initializing [ 6757.445316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.463828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6762.496679] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6774.153554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.258570] Lustre: 208034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6774.346334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6774.353924] Lustre: Skipped 1 previous similar message [ 6774.458366] Lustre: lustre-MDT0001: new disk, initializing [ 6774.667508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6774.685935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6779.303911] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6788.970835] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6794.097975] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6801.394806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6801.714654] Lustre: lustre-OST0000: new disk, initializing [ 6801.720597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6801.725986] Lustre: 210007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6808.921769] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6810.715528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6810.727676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6810.860537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6819.873290] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6824.582455] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6824.820967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6830.156861] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6830.427331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6836.708019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.711470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.720630] Lustre: Skipped 2 previous similar messages [ 6836.725271] LustreError: Skipped 1 previous similar message [ 6836.734447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6836.737723] Lustre: Skipped 3 previous similar messages [ 6838.678460] Lustre: server umount lustre-OST0000 complete [ 6857.184273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6857.724384] Lustre: server umount lustre-MDT0000 complete [ 6861.284368] LustreError: 208046:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.320667] LustreError: 208046:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 14 previous similar messages [ 6863.229270] LustreError: 210005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863071 with bad export cookie 17366323752605194804 [ 6863.230096] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6863.248521] LustreError: 210005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6863.869093] Lustre: server umount lustre-MDT0001 complete [ 6873.529374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.168746] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6889.004803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.572829] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6908.189503] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6916.748799] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6925.414144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.196688] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6942.146660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6947.503569] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6947.687535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6953.004731] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6953.340321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6959.590500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6959.595806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.607366] LustreError: Skipped 1 previous similar message [ 6959.637369] Lustre: Skipped 4 previous similar messages [ 6961.964065] Lustre: server umount lustre-OST0000 complete [ 6971.199496] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6976.302562] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7005.503918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7012.336509] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7020.465044] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.301564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7024.314445] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7024.327359] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7024.347305] Lustre: Skipped 1 previous similar message [ 7024.413590] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 7029.733551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7029.743255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.750472] LustreError: Skipped 1 previous similar message [ 7029.760045] Lustre: Skipped 9 previous similar messages [ 7034.829719] Lustre: server umount lustre-OST0000 complete [ 7044.947617] Lustre: server umount lustre-MDT0000 complete [ 7049.153946] LustreError: 214071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863257 with bad export cookie 17366323752605195686 [ 7049.158303] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7049.170926] LustreError: 214071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7049.717406] Lustre: server umount lustre-MDT0001 complete [ 7059.158977] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:41:05 (1787863265) [ 7066.658060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.217376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7067.226466] Lustre: Skipped 6 previous similar messages [ 7071.797379] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7079.957392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.848783] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7093.191926] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7101.141654] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7108.554920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.290419] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7123.802705] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7129.019946] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7129.226753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7133.639473] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7133.921337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7140.747915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7140.968198] Lustre: lustre-OST0001: new disk, initializing [ 7140.973591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7140.979332] Lustre: 221570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7147.748491] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7151.268848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7151.276423] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7151.420119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7158.307269] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7163.477182] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7163.692881] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7168.966587] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7169.210791] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7172.076343] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.077321] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.130269] Lustre: Skipped 4 previous similar messages [ 7177.964841] Lustre: server umount lustre-OST0001 complete [ 7182.321191] LustreError: 220020:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.354085] LustreError: 220020:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 7186.441070] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7186.728605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7191.910568] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7192.114427] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7221.719810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.372681] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7227.382520] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7227.392374] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7227.405149] Lustre: Skipped 1 previous similar message [ 7228.916669] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7236.450646] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7237.953513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7253.984315] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7254.065725] Lustre: server umount lustre-OST0001 complete [ 7277.557299] Lustre: server umount lustre-OST0000 complete [ 7287.866513] Lustre: server umount lustre-MDT0000 complete [ 7291.950913] LustreError: 218178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863500 with bad export cookie 17366323752605196652 [ 7291.960395] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.973355] LustreError: 218178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7298.403786] Lustre: server umount lustre-MDT0001 complete [ 7322.852474] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:45:29 (1787863529) [ 7329.586113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.865701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7329.892645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7334.385189] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7342.475946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.535723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7342.566764] Lustre: 226815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7347.081712] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7355.133622] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7361.316703] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7368.945752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7369.324137] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7376.593462] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7384.117641] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7393.301260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.381658] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7400.141536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7407.708849] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7409.174770] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7413.146265] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7413.452724] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7417.754329] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7417.955887] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7423.809777] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c9410da4000.ost_server_uuid 50 [ 7425.595466] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c9410da4000.ost_server_uuid in IDLE state after 0 sec [ 7426.815366] Lustre: Permanently deactivating lustre-OST0001 [ 7426.837220] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7430.984760] Lustre: Permanently reactivating lustre-OST0001 [ 7431.003698] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7431.009391] Lustre: Skipped 2 previous similar messages [ 7436.277045] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7436.290586] LustreError: Skipped 3 previous similar messages [ 7436.296508] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.314851] Lustre: Skipped 7 previous similar messages [ 7436.323785] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7436.330688] Lustre: Skipped 16 previous similar messages [ 7449.060125] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7449.149259] Lustre: server umount lustre-OST0001 complete [ 7458.751128] Lustre: server umount lustre-OST0000 complete [ 7468.903299] Lustre: server umount lustre-MDT0000 complete [ 7472.043569] LustreError: 229685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863680 with bad export cookie 17366323752605197933 [ 7472.049129] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.056962] LustreError: 229685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7472.317978] Lustre: server umount lustre-MDT0001 complete [ 7497.062656] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:48:23 (1787863703) [ 7504.928541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.278832] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7505.313622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7505.323745] Lustre: Skipped 2 previous similar messages [ 7510.387501] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7519.448807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.523950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7519.550854] Lustre: 233510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7524.510896] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7532.368945] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7538.272808] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7544.632871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.838569] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7544.867115] Lustre: Permanently deactivating lustre-OST0000 [ 7544.889859] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7551.771643] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7553.056928] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7559.943713] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7567.635654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7567.721671] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7574.412105] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7577.819067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7581.881907] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7583.607331] Lustre: Permanently reactivating lustre-OST0000 [ 7583.632779] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7583.638219] Lustre: Skipped 2 previous similar messages [ 7584.491891] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7584.493714] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp 000000006a986068, cur 1787863792 deadline 1787863791 last 1787863761 [ 7584.503474] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7584.527642] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7584.592931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7598.636310] Lustre: server umount lustre-OST0001 complete [ 7608.758824] Lustre: server umount lustre-OST0000 complete [ 7619.992538] Lustre: server umount lustre-MDT0000 complete [ 7624.225506] LustreError: 233501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863832 with bad export cookie 17366323752605199571 [ 7624.235864] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7624.243501] LustreError: 233501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7624.634456] Lustre: server umount lustre-MDT0001 complete [ 7638.356390] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7642.102528] Key type lgssc unregistered [ 7642.454353] LNet: 238390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7642.465383] LNetError: 238390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7642.495790] LNet: Removed LNI 192.168.204.154@tcp [ 7643.641859] Key type .llcrypt unregistered [ 7643.645538] Key type ._llcrypt unregistered [ 7669.380227] Key type ._llcrypt registered [ 7669.388958] Key type .llcrypt registered [ 7669.553986] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7670.680873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7670.711972] alg: No test for adler32 (adler32-zlib) [ 7671.780800] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 7672.026054] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7673.696881] Key type lgssc registered [ 7675.007919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7686.361117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7686.384806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.933864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.318333] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7697.380880] LustreError: 239860:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.406242] LustreError: 239860:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7698.401410] LustreError: 239861:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7702.153161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.255773] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7702.274793] Lustre: 239848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7702.283481] Lustre: Permanently deactivating lustre-MDT0001 [ 7702.295403] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7702.568553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7704.965264] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7704.976068] LustreError: 239892:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7704.987926] LustreError: 239892:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7705.005383] LustreError: 239892:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7707.301133] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7715.630717] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7720.972678] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7729.242269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7729.604635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.676585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7733.672261] LustreError: 241700:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.778725] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7738.849680] LustreError: 241702:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.979446] LustreError: 242101:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7746.431529] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7747.089543] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7752.180320] LustreError: 242101:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7752.214317] LustreError: 242101:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 7757.121513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.364507] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.443164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7762.453613] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7764.386651] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7773.417506] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7778.153613] Lustre: Permanently reactivating lustre-MDT0001 [ 7778.206706] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7778.212746] Lustre: Skipped 2 previous similar messages [ 7784.397666] Lustre: lustre-MDT0001: Client aabeff65-216a-4f10-b8b9-fe4f449fb941 (at 192.168.204.54@tcp) reconnecting [ 7791.715777] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7793.351931] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7800.099449] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7800.472130] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7805.994526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7806.312666] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7809.347726] Lustre: Permanently deactivating lustre-MDT0001 [ 7809.370173] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7809.375973] Lustre: Skipped 2 previous similar messages [ 7816.293348] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7824.256866] Lustre: server umount lustre-MDT0000 complete [ 7825.379746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7825.388197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.401234] LustreError: 239860:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.413389] LustreError: 239860:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7825.900240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.915691] Lustre: Skipped 2 previous similar messages [ 7827.619693] LustreError: 239840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864035 with bad export cookie 18316676006976975332 [ 7827.621859] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7827.633414] LustreError: 239840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7827.946321] Lustre: server umount lustre-MDT0001 complete [ 7831.616934] Lustre: server umount lustre-OST0000 complete [ 7842.342402] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:54:08 (1787864048) [ 7851.969288] Lustre: server umount lustre-OST0001 complete [ 7859.850156] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7870.129429] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7909.687580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.982208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7909.991470] Lustre: Skipped 2 previous similar messages [ 7910.028169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7910.100347] Lustre: lustre-MDT0000: new disk, initializing [ 7910.223990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.248147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7915.643933] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7927.121991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.282802] Lustre: 248119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7927.343254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7927.347457] Lustre: Skipped 1 previous similar message [ 7927.425925] Lustre: lustre-MDT0001: new disk, initializing [ 7927.570745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7927.601226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7927.609806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7932.457158] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7942.126137] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7948.359233] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7955.443240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.718365] Lustre: lustre-OST0000: new disk, initializing [ 7955.728582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7955.742445] Lustre: 250087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7955.837208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.690141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7961.702620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7961.819155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7962.312163] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7971.518880] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7973.681563] LustreError: 248126:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7981.540801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.707794] LustreError: 250069:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.716038] LustreError: 250069:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 7981.731694] Lustre: lustre-OST0001: new disk, initializing [ 7981.734595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7981.741671] Lustre: 251202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7983.746251] LustreError: 248126:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7983.779783] LustreError: 248124:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7988.591129] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7991.873668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7991.882149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7991.941819] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7993.872383] LustreError: 248124:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7997.858591] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7999.776889] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8002.035483] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.044948] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8002.051015] Lustre: Skipped 1 previous similar message [ 8002.057987] Lustre: Skipped 1 previous similar message [ 8007.145732] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8007.151705] Lustre: Skipped 2 previous similar messages [ 8012.265613] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8012.275637] Lustre: Skipped 2 previous similar messages [ 8016.353242] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8016.474219] Lustre: server umount lustre-OST0001 complete [ 8033.765570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.770151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.774153] LustreError: Skipped 1 previous similar message [ 8033.789244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8033.797544] Lustre: Skipped 2 previous similar messages [ 8037.869197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8037.880200] Lustre: Skipped 1 previous similar message [ 8039.839519] Lustre: server umount lustre-OST0000 complete [ 8045.540583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8045.554511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.573804] Lustre: Skipped 1 previous similar message [ 8048.101339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.110153] Lustre: Skipped 2 previous similar messages [ 8049.866701] Lustre: server umount lustre-MDT0000 complete [ 8053.224189] LustreError: 251480:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8053.250688] LustreError: 251480:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 12 previous similar messages [ 8053.690505] LustreError: 248109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864261 with bad export cookie 18316676006976978139 [ 8053.698066] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8053.711886] LustreError: 248109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8060.130545] Lustre: server umount lustre-MDT0001 complete [ 8069.411716] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8072.496492] Key type lgssc unregistered [ 8072.753548] LNet: 253239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8072.763139] LNetError: 253239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8072.786897] LNet: Removed LNI 192.168.204.154@tcp [ 8073.734234] Key type .llcrypt unregistered [ 8073.736230] Key type ._llcrypt unregistered [ 8109.646692] Key type ._llcrypt registered [ 8109.649448] Key type .llcrypt registered [ 8109.729248] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8110.515852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8110.620151] alg: No test for adler32 (adler32-zlib) [ 8111.638533] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8111.883236] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8113.552194] Key type lgssc registered [ 8114.423058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8120.745212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8120.765499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.026993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8122.050208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8122.156336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.145407] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8133.778479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.911180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8133.946552] Lustre: 255427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8134.187418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8138.485937] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8144.492070] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8149.435505] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8156.317805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8156.651316] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8156.849608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.825711] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8167.042136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8169.709450] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8223.202080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8223.207133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.217207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8223.719628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8223.727615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.739932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8226.555270] Lustre: server umount lustre-OST0000 complete [ 8230.875370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8243.074261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8243.455893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8250.521365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8259.271715] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8263.154117] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8263.170248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8263.171901] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8263.200481] Lustre: Skipped 1 previous similar message [ 8304.116093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8304.120300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8304.125120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8304.128638] LustreError: Skipped 1 previous similar message [ 8304.159228] Lustre: Skipped 1 previous similar message [ 8306.377474] Lustre: server umount lustre-OST0000 complete [ 8313.314983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8313.328418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8313.337461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.342641] Lustre: Skipped 1 previous similar message [ 8316.845957] Lustre: server umount lustre-MDT0000 complete [ 8320.935866] LustreError: 255420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864529 with bad export cookie 16242664152638819079 [ 8320.937262] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.945837] LustreError: 255420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8321.363563] Lustre: server umount lustre-MDT0001 complete [ 8332.640510] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8335.936450] Key type lgssc unregistered [ 8336.264109] LNet: 260862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8336.281095] LNetError: 260862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8337.323785] LNet: Removed LNI 192.168.204.154@tcp [ 8338.283238] Key type .llcrypt unregistered [ 8338.285835] Key type ._llcrypt unregistered [ 8367.180560] Key type ._llcrypt registered [ 8367.183295] Key type .llcrypt registered [ 8367.297518] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8368.889661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8368.976380] alg: No test for adler32 (adler32-zlib) [ 8370.049024] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8370.274843] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8371.928193] Key type lgssc registered [ 8373.185310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8381.147830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8381.170570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.555427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8386.521463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8392.675507] LustreError: 262280:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8392.694566] LustreError: 262280:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8396.181054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8396.845229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8401.822965] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8411.007768] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8418.642084] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8425.717615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8426.103894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8427.124331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8432.213470] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8439.648679] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8467.425527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8467.430751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.450873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8467.473554] Lustre: Skipped 1 previous similar message [ 8472.022834] Lustre: server umount lustre-OST0000 complete [ 8477.665764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8477.687475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8477.696038] Lustre: Skipped 2 previous similar messages [ 8482.788114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8482.802198] Lustre: Skipped 1 previous similar message [ 8483.451934] Lustre: server umount lustre-MDT0000 complete [ 8486.881844] LustreError: 262275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.903089] LustreError: 262275:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8487.626775] LustreError: 262262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864695 with bad export cookie 15961119078159879242 [ 8487.636688] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8487.645204] LustreError: 262262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8488.010329] Lustre: server umount lustre-MDT0001 complete [ 8498.616980] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8501.602453] Key type lgssc unregistered [ 8501.839879] LNet: 266590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8501.847857] LNetError: 266590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8501.865649] LNet: Removed LNI 192.168.204.154@tcp [ 8502.703336] Key type .llcrypt unregistered [ 8502.704911] Key type ._llcrypt unregistered [ 8521.500278] Key type ._llcrypt registered [ 8521.504662] Key type .llcrypt registered [ 8521.617887] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8522.690332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8522.826083] alg: No test for adler32 (adler32-zlib) [ 8523.918575] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8524.182342] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8525.832223] Key type lgssc registered [ 8526.936440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8534.119208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8534.159297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.631515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.221371] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8545.766091] LustreError: 267813:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8545.778069] LustreError: 267813:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8550.281823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.672781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8556.080684] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8563.923893] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8571.162313] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8578.614444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8579.046656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.099895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8585.354631] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8593.994165] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8610.786206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8610.797188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8610.810534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8610.829342] Lustre: Skipped 1 previous similar message [ 8614.455118] Lustre: server umount lustre-OST0000 complete [ 8621.029694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.039933] Lustre: Skipped 1 previous similar message [ 8621.046405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8621.055931] Lustre: Skipped 1 previous similar message [ 8624.471693] Lustre: server umount lustre-MDT0000 complete [ 8626.145637] LustreError: 267809:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.176936] LustreError: 267809:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8628.122152] LustreError: 268547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864836 with bad export cookie 8309125421921453807 [ 8628.130132] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.131462] LustreError: 268547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8628.620917] Lustre: server umount lustre-MDT0001 complete [ 8641.052962] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8644.053205] Key type lgssc unregistered [ 8644.390335] LNet: 271495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8644.415881] LNetError: 271495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8644.435591] LNet: Removed LNI 192.168.204.154@tcp [ 8645.370856] Key type .llcrypt unregistered [ 8645.373971] Key type ._llcrypt unregistered [ 8674.390720] Key type ._llcrypt registered [ 8674.396244] Key type .llcrypt registered [ 8674.623780] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8675.814877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8675.880597] alg: No test for adler32 (adler32-zlib) [ 8677.272703] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8677.990948] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8679.920244] Key type lgssc registered [ 8682.009305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8691.136285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8691.192783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.547250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.443598] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8701.933852] LustreError: 272916:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8701.949706] LustreError: 272916:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8705.578578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.932382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8710.400141] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8717.598959] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8724.268777] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8731.470641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.774282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.906590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8738.507084] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8747.200510] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8776.165539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8776.165960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.176690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.192881] Lustre: Skipped 1 previous similar message [ 8781.288757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8781.302729] Lustre: Skipped 1 previous similar message [ 8782.077565] Lustre: server umount lustre-OST0000 complete [ 8787.941341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8787.962838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.996475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8788.001121] Lustre: Skipped 1 previous similar message [ 8791.525485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8791.524885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8791.551026] Lustre: Skipped 1 previous similar message [ 8796.645871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.652799] Lustre: Skipped 1 previous similar message [ 8801.251906] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8801.488154] Lustre: server umount lustre-MDT0000 complete [ 8801.763628] LustreError: 272910:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8801.790966] LustreError: 272910:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8805.414595] LustreError: 274758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865013 with bad export cookie 3607574615837925757 [ 8805.420350] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8805.428279] LustreError: 274758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8805.836172] Lustre: server umount lustre-MDT0001 complete [ 8815.325583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8818.656418] Key type lgssc unregistered [ 8818.986529] LNet: 277224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8818.999451] LNetError: 277224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8819.021492] LNet: Removed LNI 192.168.204.154@tcp [ 8819.850422] Key type .llcrypt unregistered [ 8819.853818] Key type ._llcrypt unregistered [ 8836.584138] Key type ._llcrypt registered [ 8836.586486] Key type .llcrypt registered [ 8836.721350] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8837.766951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8837.794092] alg: No test for adler32 (adler32-zlib) [ 8838.858159] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8839.050495] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8840.720480] Key type lgssc registered [ 8841.794630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.548399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.568488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.172950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8855.497575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8861.683869] LustreError: 278462:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.710698] LustreError: 278462:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8863.871812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.296447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.807738] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8877.125720] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8883.094694] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8890.994287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.374830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.515428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8898.723564] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8905.996821] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8922.081792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8922.095883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.111349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8925.157859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8925.159156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8925.159683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8926.409762] Lustre: server umount lustre-OST0000 complete [ 8930.789299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8930.795513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8930.807992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.825296] Lustre: Skipped 1 previous similar message [ 8935.395484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8935.395571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.717371] Lustre: server umount lustre-MDT0000 complete [ 8940.457933] LustreError: 278428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865148 with bad export cookie 16428160335952784373 [ 8940.477939] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8940.481021] LustreError: 278428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8940.758888] Lustre: server umount lustre-MDT0001 complete [ 8950.405117] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8952.957337] Key type lgssc unregistered [ 8953.278806] LNet: 282166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.286320] LNetError: 282166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8953.302328] LNet: Removed LNI 192.168.204.154@tcp [ 8954.056526] Key type .llcrypt unregistered [ 8954.060974] Key type ._llcrypt unregistered [ 8965.503490] Key type ._llcrypt registered [ 8965.506120] Key type .llcrypt registered [ 8965.612160] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8981.458269] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8992.666531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8993.468547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8993.489111] alg: No test for adler32 (adler32-zlib) [ 8994.537318] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 8994.818326] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8996.505429] Key type lgssc registered [ 8997.584660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9029.906633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9029.940969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.277146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9031.306716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9031.416827] Lustre: lustre-MDT0000: new disk, initializing [ 9031.613465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.649555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9036.013195] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9046.516150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.659552] Lustre: 285702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9046.747771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9046.752337] Lustre: Skipped 1 previous similar message [ 9046.830419] Lustre: lustre-MDT0001: new disk, initializing [ 9047.018275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9047.068899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9047.076613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9052.424717] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9062.505839] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9069.065928] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9076.378741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9076.638358] Lustre: lustre-OST0000: new disk, initializing [ 9076.641394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9076.645813] Lustre: 287670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9076.737936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.487152] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9084.534331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9084.550666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9084.737822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9094.539340] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9099.886934] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9100.108471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9104.429846] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9104.689122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9110.499032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9110.504667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9110.508481] LustreError: Skipped 1 previous similar message [ 9110.509902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9110.530735] Lustre: Skipped 1 previous similar message [ 9112.416429] Lustre: server umount lustre-OST0000 complete [ 9115.618034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.620137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9115.626698] Lustre: Skipped 1 previous similar message [ 9115.632374] Lustre: Skipped 1 previous similar message [ 9120.738194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9120.756136] Lustre: Skipped 2 previous similar messages [ 9121.551387] Lustre: server umount lustre-MDT0000 complete [ 9125.635829] LustreError: 286506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865333 with bad export cookie 1116337101039308168 [ 9125.639805] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9125.646528] LustreError: 286506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9126.281493] Lustre: server umount lustre-MDT0001 complete [ 9136.703147] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:15:42 (1787865342) [ 9143.546470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9144.014805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.300595] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9149.425144] LustreError: 290083:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9149.453327] LustreError: 290083:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9154.531744] LustreError: 290082:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9155.558953] LustreError: 290083:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.090969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.418376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9161.588911] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9168.424113] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9174.048282] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9182.361518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9182.822777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9190.147602] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9198.643644] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9212.900082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.906090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9212.916145] Lustre: Skipped 1 previous similar message [ 9218.022872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9218.027245] Lustre: Skipped 1 previous similar message [ 9219.032849] Lustre: server umount lustre-OST0000 complete [ 9223.144422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9223.159803] Lustre: Skipped 1 previous similar message [ 9228.263395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.276156] Lustre: Skipped 4 previous similar messages [ 9229.344718] Lustre: server umount lustre-MDT0000 complete [ 9233.095416] LustreError: 290062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865441 with bad export cookie 1116337101039309050 [ 9233.099273] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9233.107900] LustreError: 290062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9233.533616] Lustre: server umount lustre-MDT0001 complete [ 9243.884311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9246.642571] Key type lgssc unregistered [ 9246.909728] LNet: 293642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9246.916983] LNetError: 293642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9246.934209] LNet: Removed LNI 192.168.204.154@tcp [ 9247.862656] Key type .llcrypt unregistered [ 9247.868135] Key type ._llcrypt unregistered [ 9272.198303] Key type ._llcrypt registered [ 9272.200047] Key type .llcrypt registered [ 9272.358463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9286.011767] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9287.041908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9287.057702] alg: No test for adler32 (adler32-zlib) [ 9288.131118] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 9288.443390] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9290.136182] Key type lgssc registered [ 9291.191297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9340.429947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9340.461922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.711331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9341.752914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9341.823396] Lustre: lustre-MDT0000: new disk, initializing [ 9341.912953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.928414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9346.418249] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9358.391202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9358.573556] Lustre: 297469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9358.674458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9358.682701] Lustre: Skipped 1 previous similar message [ 9358.798525] Lustre: lustre-MDT0001: new disk, initializing [ 9358.932551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9359.061058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9359.081907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9364.453394] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9375.007563] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9382.156447] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9390.551434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9390.975291] Lustre: lustre-OST2710: new disk, initializing [ 9390.980420] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9390.987505] Lustre: 299438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9391.114864] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9398.332480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9398.351716] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9398.439617] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9398.490238] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9409.512366] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9417.000820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.193394] Lustre: lustre-OST03e8: new disk, initializing [ 9417.200646] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9417.207929] Lustre: 300501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9417.299817] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9423.545802] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9424.502578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9424.532825] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9424.671138] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9431.574963] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9439.780883] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9439.982354] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9444.279236] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9444.475697] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9449.038297] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9449.241549] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9454.004739] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9454.268943] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9458.662440] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.675975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9458.684756] Lustre: Skipped 1 previous similar message [ 9458.712163] Lustre: Skipped 3 previous similar messages [ 9462.531503] Lustre: server umount lustre-MDT0000 complete [ 9465.964117] LustreError: 297460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865674 with bad export cookie 8658627680412139602 [ 9465.975992] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9466.326567] Lustre: server umount lustre-MDT0001 complete [ 9480.245741] Lustre: server umount lustre-OST2710 complete [ 9493.878293] Lustre: server umount lustre-OST03e8 complete [ 9510.595316] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9520.596814] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9555.485438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9555.696899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9555.722905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9555.834814] Lustre: lustre-MDT0000: new disk, initializing [ 9555.915924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.953157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9560.789158] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9572.423371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9572.548067] Lustre: 305751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9572.604349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9572.608846] Lustre: Skipped 1 previous similar message [ 9572.695719] Lustre: lustre-MDT0001: new disk, initializing [ 9572.782546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9572.842500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9572.857918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9577.424032] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9588.306166] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9594.454498] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9601.124790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9601.533861] Lustre: lustre-OST0000: new disk, initializing [ 9601.539865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9601.550692] Lustre: 307720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9601.667843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.788619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9608.803891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9608.905825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9609.047649] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9618.878196] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9624.532989] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9624.782787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9629.849556] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9630.023259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9634.785574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9634.793709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.801353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9634.807922] Lustre: Skipped 3 previous similar messages [ 9639.914209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9639.927826] Lustre: Skipped 2 previous similar messages [ 9645.053868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9645.067493] Lustre: Skipped 1 previous similar message [ 9646.048159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9646.194557] Lustre: server umount lustre-OST0000 complete [ 9654.753750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9654.760954] LustreError: Skipped 1 previous similar message [ 9654.766689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.784066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9656.784554] Lustre: server umount lustre-MDT0000 complete [ 9660.387508] LustreError: 305762:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.398409] LustreError: 305762:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 9660.839197] LustreError: 305742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865868 with bad export cookie 8658627680412141282 [ 9660.852493] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9660.855956] LustreError: 305742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9661.182269] Lustre: server umount lustre-MDT0001 complete [ 9670.221663] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:24:36 (1787865876) [ 9672.021435] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9673.712286] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:24:40 (1787865880) [ 9676.869492] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9702.233102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9702.489334] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9702.509972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9702.635183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.127132] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9715.226278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9715.345717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9715.398331] Lustre: 311580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9720.660779] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9742.318780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.464818] Lustre: Denying initial registration attempt from nid 192.168.204.154@tcp, specified as failover [ 9742.471075] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9742.479863] LustreError: 313034:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9742.484674] LustreError: 313034:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9742.489187] LustreError: 313034:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9742.539698] Lustre: server umount lustre-OST0000 complete [ 9742.542247] LustreError: 313034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9749.986946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9749.988433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9750.002143] Lustre: Skipped 2 previous similar messages [ 9750.006345] Lustre: Skipped 2 previous similar messages [ 9755.668074] Lustre: server umount lustre-MDT0000 complete [ 9759.579688] LustreError: 311574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865967 with bad export cookie 8658627680412142171 [ 9759.581407] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9759.602533] LustreError: 311574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9760.018690] Lustre: server umount lustre-MDT0001 complete [ 9771.853918] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9774.684566] Key type lgssc unregistered [ 9774.956456] LNet: 314158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9774.966905] LNetError: 314158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9774.991284] LNet: Removed LNI 192.168.204.154@tcp [ 9775.863476] Key type .llcrypt unregistered [ 9775.867976] Key type ._llcrypt unregistered [ 9786.899237] Key type ._llcrypt registered [ 9786.901836] Key type .llcrypt registered [ 9787.022939] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9787.862719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9787.889402] alg: No test for adler32 (adler32-zlib) [ 9788.998635] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 9789.308988] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9791.024231] Key type lgssc registered [ 9792.117873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9844.423557] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9854.051899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9854.072055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.344546] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9855.367531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9855.599639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.197822] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9869.012304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.151203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9869.203220] Lustre: 316923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9869.474692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9874.272166] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9883.188537] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9890.521095] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9912.040093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9912.212504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9912.372257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.224678] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9927.795322] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9933.281370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9933.293231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9933.313560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9933.804149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9933.811294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9933.826150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9936.420767] Lustre: server umount lustre-OST0000 complete [ 9941.474339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9941.481267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.511667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9944.034505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9944.048518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9946.934605] Lustre: server umount lustre-MDT0000 complete [ 9949.153609] LustreError: 316935:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.164135] LustreError: 316935:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9950.866187] LustreError: 318801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866158 with bad export cookie 13046747800834802052 [ 9950.870542] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.879028] LustreError: 318801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9951.268688] Lustre: server umount lustre-MDT0001 complete [ 9965.616684] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9968.156628] Key type lgssc unregistered [ 9968.483768] LNet: 320503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9968.492225] LNetError: 320503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9968.528933] LNet: Removed LNI 192.168.204.154@tcp [ 9969.479855] Key type .llcrypt unregistered [ 9969.482215] Key type ._llcrypt unregistered [ 9995.738393] Key type ._llcrypt registered [ 9995.748612] Key type .llcrypt registered [ 9995.905396] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9997.263185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9997.294560] alg: No test for adler32 (adler32-zlib) [ 9998.406677] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 9998.641873] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10000.392215] Key type lgssc registered [10001.791216] Lustre: Echo OBD driver; http://www.lustre.org/ [10010.549930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10010.580331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.049882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10016.979466] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10022.372327] LustreError: 321921:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.384160] LustreError: 321921:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10026.454447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10026.903370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10031.774373] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10039.521436] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10045.783264] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10054.193912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10054.542790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10061.006745] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10069.501236] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10081.266091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.270565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.289342] Lustre: Skipped 2 previous similar messages [10081.306015] Lustre: Skipped 1 previous similar message [10082.197374] Lustre: server umount lustre-MDT0000 complete [10082.747368] LustreError: 323927:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.373545] LustreError: 321916:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.406616] LustreError: 321916:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10086.757896] LustreError: 321903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866294 with bad export cookie 1747643220694267453 [10086.761939] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10086.772369] LustreError: 321903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10087.855509] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [10087.864937] Lustre: Skipped 1 previous similar message [10091.492202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.493433] LustreError: 321916:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10091.507857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10091.543977] LustreError: 321916:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10096.610680] LustreError: 321918:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10096.613474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10096.629513] LustreError: 321918:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10096.665390] Lustre: Skipped 1 previous similar message [10101.218113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10101.545176] Lustre: server umount lustre-MDT0001 complete [10108.638385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10118.028136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.838686] LustreError: 323754:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10132.852913] LustreError: 323754:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10132.923486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10133.002277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10136.735570] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10145.109887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10145.496760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10149.791552] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10151.921145] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10158.256944] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10160.265399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10167.049189] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10168.916504] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10176.490309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10176.501062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.516962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10176.531774] Lustre: Skipped 3 previous similar messages [10180.384263] Lustre: server umount lustre-OST0000 complete [10186.209475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10186.216389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.231770] Lustre: Skipped 1 previous similar message [10186.240200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.246456] Lustre: Skipped 2 previous similar messages [10191.020068] Lustre: server umount lustre-MDT0000 complete [10192.872532] LustreError: 325518:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.891475] LustreError: 325518:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10194.685902] LustreError: 321903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866402 with bad export cookie 1747643220694274628 [10194.692453] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10194.696686] LustreError: 321903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10195.035432] Lustre: server umount lustre-MDT0001 complete [10207.446682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10210.611284] Key type lgssc unregistered [10210.961037] LNet: 328190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10210.972959] LNetError: 328190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10211.015135] LNet: Removed LNI 192.168.204.154@tcp [10211.854615] Key type .llcrypt unregistered [10211.856832] Key type ._llcrypt unregistered [10241.914897] Key type ._llcrypt registered [10241.917410] Key type .llcrypt registered [10242.001468] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10243.031547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10243.111273] alg: No test for adler32 (adler32-zlib) [10244.214561] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [10244.419659] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10246.072349] Key type lgssc registered [10247.478653] Lustre: Echo OBD driver; http://www.lustre.org/ [10256.422049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10256.437882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10257.812930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10262.507857] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10267.107674] LustreError: 329609:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.133097] LustreError: 329609:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10272.228712] LustreError: 329610:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10272.287383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10272.888400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10277.813966] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10285.655658] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10292.133078] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10297.701978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10297.707996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10297.720765] Lustre: Skipped 1 previous similar message [10297.736385] Lustre: Skipped 1 previous similar message [10301.884419] Lustre: server umount lustre-MDT0000 complete [10302.944830] LustreError: 329610:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.958453] LustreError: 329610:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10305.582167] LustreError: 329590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866513 with bad export cookie 17691905109025380494 [10305.599160] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10305.820218] Lustre: server umount lustre-MDT0001 complete [10321.785982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.154387] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10322.211540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10322.467572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10327.400865] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10335.888526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.002891] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10336.053437] Lustre: 332196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10336.333909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10341.228739] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10348.185708] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10353.060559] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10362.667333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10362.923848] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10362.937149] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10362.956408] LustreError: 334079:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10362.964294] LustreError: 334079:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10362.973217] LustreError: 334079:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10363.050286] Lustre: server umount lustre-OST0000 complete [10363.055704] LustreError: 334079:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10374.895822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10375.135286] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10375.376623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10381.008457] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10384.556962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10389.727139] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10398.674114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10398.734918] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10398.801416] Lustre: lustre-OST0001: new disk, initializing [10398.805330] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10398.812111] Lustre: 335623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10398.926115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10406.574130] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10407.524734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10407.536777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10407.666404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10415.224769] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10423.271026] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10423.292284] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10423.312229] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10423.316539] Lustre: Skipped 1 previous similar message [10425.058729] Lustre: server umount lustre-OST0001 complete [10428.402988] LustreError: 334546:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.428574] LustreError: 334546:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10430.433465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10430.449154] LustreError: Skipped 1 previous similar message [10430.465243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.487449] Lustre: Skipped 1 previous similar message [10430.494343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10433.510800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10433.523248] LustreError: 336580:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.525447] Lustre: Skipped 2 previous similar messages [10433.556784] LustreError: 336580:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10435.704938] Lustre: server umount lustre-OST0000 complete [10443.746927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10443.752855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10443.763423] LustreError: Skipped 1 previous similar message [10443.764665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10443.786387] Lustre: Skipped 2 previous similar messages [10446.349595] Lustre: server umount lustre-MDT0000 complete [10448.870306] LustreError: 332204:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.894880] LustreError: 332204:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10450.914210] LustreError: 332191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866659 with bad export cookie 17691905109025381047 [10450.929899] LustreError: 332191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10450.936358] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10451.348916] Lustre: server umount lustre-MDT0001 complete [10462.893342] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10465.936833] Key type lgssc unregistered [10466.261050] LNet: 337584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10466.288122] LNetError: 337584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10466.321105] LNet: Removed LNI 192.168.204.154@tcp [10467.454219] Key type .llcrypt unregistered [10467.456450] Key type ._llcrypt unregistered [10541.528088] Key type ._llcrypt registered [10541.530562] Key type .llcrypt registered [10541.629520] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [10558.480233] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10559.992653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10560.055899] alg: No test for adler32 (adler32-zlib) [10561.168555] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [10561.451363] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10563.152211] Key type lgssc registered [10564.405824] Lustre: Echo OBD driver; http://www.lustre.org/ [10600.653974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10600.699946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.909893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10601.966478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10602.056816] Lustre: lustre-MDT0000: new disk, initializing [10602.153531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10602.174189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10606.948990] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10617.803788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.928966] Lustre: 342521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10618.001152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10618.005847] Lustre: Skipped 1 previous similar message [10618.082885] Lustre: lustre-MDT0001: new disk, initializing [10618.190762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10618.243654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10618.257847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10622.716725] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10632.709952] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10638.814337] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10646.158944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10646.392946] Lustre: lustre-OST0000: new disk, initializing [10646.396738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10646.406033] Lustre: 344489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10646.525632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10653.122474] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10654.300403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10654.317181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10654.465628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10662.934365] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10667.843718] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10668.023862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10671.892449] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10672.033878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10675.174355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10675.176122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.211346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10680.179225] Lustre: server umount lustre-OST0000 complete [10684.898039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10684.914581] LustreError: Skipped 1 previous similar message [10684.919033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10684.949890] Lustre: Skipped 1 previous similar message [10684.962874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10684.976173] Lustre: Skipped 1 previous similar message [10690.534765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10690.549104] Lustre: Skipped 3 previous similar messages [10695.654776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10695.657791] Lustre: Skipped 1 previous similar message [10698.211131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10698.354612] Lustre: server umount lustre-MDT0000 complete [10702.056799] LustreError: 342514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866910 with bad export cookie 4298465695856704555 [10702.070555] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10702.073420] LustreError: 342514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10702.346347] Lustre: server umount lustre-MDT0001 complete [10712.953500] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:41:59 (1787866919) [10722.933281] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:42:08 (1787866928) [10734.584453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.427600] LustreError: 347248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.467758] LustreError: 347248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10735.566728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10740.655163] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10740.710377] LustreError: 347249:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.843311] LustreError: 347248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.569865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.922678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10752.203506] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10760.019478] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10764.841902] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10771.272271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10771.645569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10777.969788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10787.048616] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10794.474666] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10796.312980] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10797.997753] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10799.468519] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10801.061498] Lustre: Failing over lustre-MDT0000 [10801.279609] Lustre: server umount lustre-MDT0000 complete [10801.634586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10801.637862] LustreError: 349017:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.647910] Lustre: Skipped 3 previous similar messages [10801.660593] LustreError: 349017:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10806.755375] LustreError: 347248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10806.777025] LustreError: 347248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10817.001883] LustreError: 347244:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.019565] LustreError: 347244:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [10820.622683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10820.778899] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10821.058921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10821.114075] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10825.631506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10826.226263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10826.258754] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10826.301948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10826.401993] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10835.753344] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10837.654924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10841.908238] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10844.392432] Lustre: Failing over lustre-MDT0000 [10844.640909] Lustre: server umount lustre-MDT0000 complete [10846.151855] LustreError: 347245:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.184294] LustreError: 347245:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10846.693414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10846.693826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10846.717639] Lustre: Skipped 2 previous similar messages [10862.049834] Lustre: 340745:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867054/real 1787867054] req@0000000080b4c194 x1874714185262336/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787867070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10862.069696] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10863.500089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.697486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10872.803993] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10876.823387] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10877.945472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10877.954914] Lustre: Skipped 2 previous similar messages [10877.986222] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10878.051669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10878.830927] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10888.513868] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10890.415899] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10898.401092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10898.411838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.415923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10898.425016] Lustre: Skipped 2 previous similar messages [10898.443306] Lustre: Skipped 1 previous similar message [10902.541476] Lustre: server umount lustre-MDT0000 complete [10903.533198] LustreError: 349490:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.553817] LustreError: 349490:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 25 previous similar messages [10905.959907] LustreError: 349090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867114 with bad export cookie 4298465695856707271 [10905.966285] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10905.971630] LustreError: 349090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10906.296977] Lustre: server umount lustre-MDT0001 complete [10920.735759] Lustre: server umount lustre-OST0000 complete [10935.960322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.154566] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10949.015259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.376424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10949.388823] Lustre: Skipped 1 previous similar message [10953.508569] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10960.173166] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10964.742580] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10970.351857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10975.895836] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10981.941530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10982.873176] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10986.979629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10986.989822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10987.000710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10987.006397] Lustre: Skipped 1 previous similar message [10989.163103] Lustre: server umount lustre-OST0000 complete [10998.302280] Lustre: server umount lustre-MDT0000 complete [11002.199793] LustreError: 353573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867210 with bad export cookie 4298465695856708013 [11002.205507] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11002.209407] LustreError: 353573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11002.761292] Lustre: server umount lustre-MDT0001 complete [11014.673164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11017.273293] Key type lgssc unregistered [11017.552898] LNet: 357124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11017.560398] LNetError: 357124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11017.574462] LNet: Removed LNI 192.168.204.154@tcp [11018.358126] Key type .llcrypt unregistered [11018.361489] Key type ._llcrypt unregistered [11049.429665] Key type ._llcrypt registered [11049.431906] Key type .llcrypt registered [11049.532823] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11050.448952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11050.575874] alg: No test for adler32 (adler32-zlib) [11051.736408] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [11052.188299] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11054.024549] Key type lgssc registered [11054.951029] Lustre: Echo OBD driver; http://www.lustre.org/ [11062.573497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11062.593671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.197733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11068.988656] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11074.533964] LustreError: 358640:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.567121] LustreError: 358640:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11079.190147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.660665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11083.345979] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11091.825342] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11098.077527] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11104.650134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11104.982781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11110.120659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11110.719247] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11118.036971] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11130.857141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11130.868640] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11130.875649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11130.910331] Lustre: Skipped 2 previous similar messages [11134.131103] Lustre: server umount lustre-MDT0000 complete [11135.970648] LustreError: 358641:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.997369] LustreError: 358641:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11138.030886] LustreError: 359375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867346 with bad export cookie 6133223249523209626 [11138.032831] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11138.040767] LustreError: 359375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11138.453305] Lustre: server umount lustre-MDT0001 complete [11152.724609] Lustre: server umount lustre-OST0000 complete [11166.006404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.427663] LustreError: 362351:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.501324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11170.511462] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11171.810725] LustreError: 362352:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.906370] LustreError: 362351:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11179.733765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.191344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11184.908939] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11192.159407] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11200.386200] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11209.092215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11209.731181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11211.819453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11217.798734] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11226.186271] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11240.425216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11240.440350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11240.441146] Lustre: Skipped 2 previous similar messages [11240.448879] Lustre: Skipped 3 previous similar messages [11244.286946] Lustre: server umount lustre-MDT0000 complete [11245.546760] LustreError: 362352:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.566268] LustreError: 362352:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11248.047752] LustreError: 362332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867456 with bad export cookie 6133223249523210739 [11248.057883] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11248.059838] LustreError: 362332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11248.352187] Lustre: server umount lustre-MDT0001 complete [11262.728477] Lustre: server umount lustre-OST0000 complete [11279.081470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.548771] LustreError: 366265:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11279.667706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11283.876255] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11292.730573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11297.088631] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11304.838571] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11311.673514] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11319.169210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11319.440128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11319.447821] Lustre: Skipped 1 previous similar message [11324.594923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11326.130583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11334.531248] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11343.330269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.345447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.348409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11343.350073] Lustre: Skipped 1 previous similar message [11346.800528] Lustre: server umount lustre-OST0000 complete [11353.572123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.572810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11353.579228] Lustre: Skipped 2 previous similar messages [11353.588208] Lustre: Skipped 2 previous similar messages [11356.758368] Lustre: server umount lustre-MDT0000 complete [11358.692637] LustreError: 366262:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.717468] LustreError: 366262:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11360.764754] LustreError: 366247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867568 with bad export cookie 6133223249523211761 [11360.786899] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11360.788311] LustreError: 366247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11361.105995] Lustre: server umount lustre-MDT0001 complete [11373.600233] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11376.175606] Key type lgssc unregistered [11376.496462] LNet: 369799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11376.507812] LNetError: 369799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11377.577732] LNet: Removed LNI 192.168.204.154@tcp [11378.543963] Key type .llcrypt unregistered [11378.546918] Key type ._llcrypt unregistered [11410.937939] Key type ._llcrypt registered [11410.946470] Key type .llcrypt registered [11411.219443] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11414.118623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11414.157730] alg: No test for adler32 (adler32-zlib) [11415.281262] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [11415.754682] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11417.624216] Key type lgssc registered [11420.006904] Lustre: Echo OBD driver; http://www.lustre.org/ [11429.867240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11429.887579] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.898877] LustreError: 371247:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11429.919933] LustreError: 371247:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11429.925990] LustreError: 371247:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11429.931512] LustreError: 371247:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11429.937690] LustreError: 371247:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11438.919991] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11438.926206] LustreError: 371644:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11438.948033] LustreError: 371644:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11438.955300] LustreError: 371644:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11438.964869] LustreError: 371644:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11438.982529] LustreError: 371644:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11461.138893] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11465.133949] Key type lgssc unregistered [11465.500462] LNet: 372652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11465.520443] LNetError: 372652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11465.545763] LNet: Removed LNI 192.168.204.154@tcp [11466.576628] Key type .llcrypt unregistered [11466.583135] Key type ._llcrypt unregistered [11488.164211] Key type ._llcrypt registered [11488.165506] Key type .llcrypt registered [11488.284175] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [11504.755272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11505.916275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11506.059961] alg: No test for adler32 (adler32-zlib) [11507.169926] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [11507.469819] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11509.176313] Key type lgssc registered [11510.602177] Lustre: Echo OBD driver; http://www.lustre.org/ [11550.320777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11550.340939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11551.648383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11551.697106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11551.841800] Lustre: lustre-MDT0000: new disk, initializing [11551.976503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11552.011850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11556.457675] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11568.253783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11568.525636] Lustre: 375814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11568.660091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11568.672129] Lustre: Skipped 1 previous similar message [11568.770973] Lustre: lustre-MDT0001: new disk, initializing [11568.999708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11569.066264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11569.081786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11574.344224] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11586.282975] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11592.888217] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11600.031584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11600.275198] Lustre: lustre-OST0000: new disk, initializing [11600.279636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11600.284866] Lustre: 377784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11600.414541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11607.851159] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11609.690220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11609.700648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11609.851271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11617.870458] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11622.540577] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11622.747923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11626.937333] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11627.067663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11630.566717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11630.576528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.581047] LustreError: Skipped 1 previous similar message [11630.586965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11630.632887] Lustre: Skipped 1 previous similar message [11635.477623] Lustre: server umount lustre-OST0000 complete [11640.804317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11640.809336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.810699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11640.818195] Lustre: Skipped 1 previous similar message [11640.824538] Lustre: Skipped 1 previous similar message [11646.050423] Lustre: server umount lustre-MDT0000 complete [11651.550680] LustreError: 376608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867859 with bad export cookie 14705723061668416814 [11651.561800] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11651.577449] LustreError: 376608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11652.054275] Lustre: server umount lustre-MDT0001 complete [11661.649037] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11663.418790] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:57:49 (1787867869) [11670.280387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.845720] LustreError: 380244:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11670.870954] LustreError: 380244:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [11670.975357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11675.302459] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11676.136494] LustreError: 380245:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.259991] LustreError: 380244:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.319801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.551217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11688.240925] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11697.807868] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11705.421650] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11713.186952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11713.585939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11719.976771] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11728.154107] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11735.479827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11735.577448] Lustre: lustre-OST0001: new disk, initializing [11735.579687] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11735.587853] Lustre: 383082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11735.671079] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11743.082364] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11744.920951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11744.933534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11745.087591] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11754.832557] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11760.621058] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11760.622566] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.630425] LustreError: Skipped 1 previous similar message [11760.642688] Lustre: Skipped 1 previous similar message [11760.642939] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11765.068233] Lustre: server umount lustre-OST0001 complete [11765.736157] LustreError: 382067:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.770213] LustreError: 382067:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11770.857132] LustreError: 383512:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.887497] LustreError: 383512:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11781.097851] LustreError: 382067:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.115309] LustreError: 382067:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11784.166124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11784.178710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11784.211661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11784.223262] Lustre: Skipped 2 previous similar messages [11789.642441] Lustre: server umount lustre-OST0000 complete [11796.450417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.466091] Lustre: Skipped 1 previous similar message [11796.485578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11796.496864] Lustre: Skipped 3 previous similar messages [11800.545548] Lustre: server umount lustre-MDT0000 complete [11801.569558] LustreError: 380972:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.582491] LustreError: 380972:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11805.069189] LustreError: 380225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868013 with bad export cookie 14705723061668417696 [11805.086467] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11805.095336] LustreError: 380225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11811.820553] Lustre: server umount lustre-MDT0001 complete [11827.256167] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [11830.651693] Key type lgssc unregistered [11831.032860] LNet: 385075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11831.042991] LNetError: 385075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.070141] LNet: Removed LNI 192.168.204.154@tcp [11832.310607] Key type .llcrypt unregistered [11832.313541] Key type ._llcrypt unregistered [11890.174883] Key type ._llcrypt registered [11890.176811] Key type .llcrypt registered [11890.297409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11892.104450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11892.174145] alg: No test for adler32 (adler32-zlib) [11893.310811] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [11893.622104] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11895.347813] Key type lgssc registered [11896.781860] Lustre: Echo OBD driver; http://www.lustre.org/ [11906.146901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11906.177211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.590571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11907.633350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11907.765187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11912.880838] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11921.793560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.897201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11921.930362] Lustre: 387886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11922.329711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11928.295453] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11936.889381] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11943.458473] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11948.003135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11948.012173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.031975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11948.041216] Lustre: Skipped 1 previous similar message [11948.068385] Lustre: Skipped 1 previous similar message [11951.940994] Lustre: server umount lustre-MDT0000 complete [11955.339758] LustreError: 387878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868163 with bad export cookie 16685262275887649502 [11955.347599] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11955.725934] Lustre: server umount lustre-MDT0001 complete [11966.486293] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:02:52 (1787868172) [11972.280440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.823135] LustreError: 390395:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.838977] LustreError: 390395:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11972.917993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11978.220049] LustreError: 390396:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.352952] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11983.338359] LustreError: 390395:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.209685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11988.506402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11995.887030] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12007.382951] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12015.362354] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12022.648313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12022.810494] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12023.100531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12029.052681] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12036.528370] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12045.968329] Lustre: Permanently deactivating lustre-OST0000 [12045.980636] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12052.528396] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12054.089630] LustreError: 393337:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12054.098866] LustreError: 393337:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12054.104668] LustreError: 393337:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12057.308764] LustreError: 393385:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12057.313991] LustreError: 393385:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12065.758341] Lustre: server umount lustre-OST0000 complete [12068.687869] LustreError: 393635:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12068.693735] LustreError: 393635:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12074.468311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12074.485202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12074.498412] Lustre: Skipped 1 previous similar message [12079.585830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12079.592384] Lustre: Skipped 1 previous similar message [12080.670941] Lustre: server umount lustre-MDT0000 complete [12084.451953] LustreError: 390376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868292 with bad export cookie 16685262275887650076 [12084.454777] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.458679] LustreError: 390376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12094.198868] Lustre: server umount lustre-MDT0001 complete [12105.296733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12109.457761] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12112.831485] LustreError: 394794:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12120.495421] Lustre: 395133:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.154@tcp [12122.599178] Lustre: server umount MGS complete [12135.493682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12135.809948] LustreError: 395859:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.834633] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12135.851455] Lustre: Skipped 1 previous similar message [12135.928397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12141.029147] LustreError: 395860:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.976472] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12151.270454] LustreError: 395859:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.297227] LustreError: 395859:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12152.651713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12152.943911] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12153.036306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12158.648586] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12167.288963] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12174.544455] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12176.370455] Lustre: Permanently reactivating lustre-OST0000 [12176.396532] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12176.402184] Lustre: Skipped 2 previous similar messages [12202.979779] LustreError: 397594:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12211.144610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12211.584614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12217.966526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12225.668299] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12233.185956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12233.200655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12233.216137] Lustre: Skipped 1 previous similar message [12233.225425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12233.911525] Lustre: server umount lustre-OST0000 complete [12239.335328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12239.367918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12239.372950] Lustre: Skipped 1 previous similar message [12248.840072] Lustre: server umount lustre-MDT0000 complete [12249.570239] LustreError: 395856:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.604327] LustreError: 395856:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12254.515014] LustreError: 395840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868462 with bad export cookie 16685262275887651133 [12254.527424] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12254.530481] LustreError: 395840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12254.948807] Lustre: server umount lustre-MDT0001 complete [12271.346469] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12275.384961] Key type lgssc unregistered [12275.779540] LNet: 400187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12275.788714] LNetError: 400187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12275.824169] LNet: Removed LNI 192.168.204.154@tcp [12276.933578] Key type .llcrypt unregistered [12276.941656] Key type ._llcrypt unregistered [12297.925750] Key type ._llcrypt registered [12297.935218] Key type .llcrypt registered [12298.178201] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [12321.054057] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12322.677784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12322.924761] alg: No test for adler32 (adler32-zlib) [12324.162367] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [12324.529339] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12326.289814] Key type lgssc registered [12328.479600] Lustre: Echo OBD driver; http://www.lustre.org/ [12368.253512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12368.284663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12369.734707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12369.782406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12369.877707] Lustre: lustre-MDT0000: new disk, initializing [12370.031227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12370.055121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12374.894265] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12384.752930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.921831] Lustre: 403344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12385.043189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12385.048500] Lustre: Skipped 1 previous similar message [12385.193662] Lustre: lustre-MDT0001: new disk, initializing [12385.409346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12385.505914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12385.527869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12391.371479] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12403.134721] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12409.661399] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12418.006183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12418.265112] Lustre: lustre-OST0000: new disk, initializing [12418.268459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12418.275990] Lustre: 405312:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12418.372228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12425.466172] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12425.893327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12425.906864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12426.050593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12436.727898] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12441.558111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12441.835215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12446.937391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12447.125735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12451.810737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12451.820433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12451.827392] LustreError: Skipped 1 previous similar message [12451.835028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12451.850060] Lustre: Skipped 1 previous similar message [12451.863824] Lustre: Skipped 1 previous similar message [12455.609167] Lustre: server umount lustre-OST0000 complete [12462.055458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.059113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12462.073832] Lustre: Skipped 1 previous similar message [12462.094415] Lustre: Skipped 1 previous similar message [12467.171861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12467.187280] Lustre: Skipped 1 previous similar message [12472.290092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12472.300288] Lustre: Skipped 1 previous similar message [12474.336184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12474.673299] Lustre: server umount lustre-MDT0000 complete [12479.116225] LustreError: 403337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868687 with bad export cookie 16421673269847806063 [12479.118646] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12479.129314] LustreError: 403337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12479.541053] Lustre: server umount lustre-MDT0001 complete [12489.911330] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12492.265686] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:11:38 (1787868698) [12499.982794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12500.691219] LustreError: 407777:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.709300] LustreError: 407777:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12500.799657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12506.081526] LustreError: 407778:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.963964] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12510.186509] LustreError: 407777:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.319402] LustreError: 407778:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.415872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.047384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12523.688606] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12532.250585] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12538.608172] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12545.741757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12546.108866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12552.104546] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12561.226131] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12563.305616] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12575.575278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12583.905559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12583.919703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12583.942859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12586.474243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12586.491825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12587.861309] Lustre: server umount lustre-OST0000 complete [12594.657385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12594.673051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.714820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12594.730461] Lustre: Skipped 2 previous similar messages [12598.133222] Lustre: server umount lustre-MDT0000 complete [12601.828531] LustreError: 410557:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12601.854229] LustreError: 410557:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12603.318480] LustreError: 407757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868811 with bad export cookie 16421673269847806945 [12603.319722] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12603.340087] LustreError: 407757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12603.748647] Lustre: server umount lustre-MDT0001 complete [12616.647175] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12619.436224] Key type lgssc unregistered [12619.793608] LNet: 411531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12619.815312] LNetError: 411531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12619.867159] LNet: Removed LNI 192.168.204.154@tcp [12620.932753] Key type .llcrypt unregistered [12620.934914] Key type ._llcrypt unregistered [12650.756233] Key type ._llcrypt registered [12650.760368] Key type .llcrypt registered [12650.860900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12651.172594] Key type .llcrypt unregistered [12651.175195] Key type ._llcrypt unregistered [12671.321714] Key type ._llcrypt registered [12671.325384] Key type .llcrypt registered [12671.486903] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12672.850479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12672.954871] alg: No test for adler32 (adler32-zlib) [12674.058116] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [12674.353099] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12676.160187] Key type lgssc registered [12677.655983] Lustre: Echo OBD driver; http://www.lustre.org/ [12687.003236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12687.027325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12688.466121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12693.197925] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12698.599189] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.631741] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12702.620923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12703.019501] LustreError: 413751:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.070987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12708.323029] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.355136] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12710.752830] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12713.453731] LustreError: 413751:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.476215] LustreError: 413751:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12713.481279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12718.562173] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.588595] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12719.828747] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12728.219558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12728.850075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12733.112829] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12744.166657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12744.176860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.199819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12744.220935] Lustre: Skipped 1 previous similar message [12748.829949] Lustre: server umount lustre-OST0000 complete [12754.241815] Lustre: server umount lustre-MDT0000 complete [12754.406079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12754.411533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.428854] Lustre: Skipped 1 previous similar message [12754.428882] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12754.461047] LustreError: 413750:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12758.891533] LustreError: 413732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868967 with bad export cookie 14171087930652738559 [12758.899459] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12758.908488] LustreError: 413732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12759.365698] Lustre: server umount lustre-MDT0001 complete [12772.823602] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12776.755756] Key type lgssc unregistered [12777.136794] LNet: 416902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12777.153105] LNetError: 416902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12777.176985] LNet: Removed LNI 192.168.204.154@tcp [12778.325351] Key type .llcrypt unregistered [12778.327622] Key type ._llcrypt unregistered [12806.370301] Key type ._llcrypt registered [12806.376672] Key type .llcrypt registered [12806.541575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12808.005806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12808.034133] alg: No test for adler32 (adler32-zlib) [12809.390414] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [12809.662939] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [12811.392245] Key type lgssc registered [12812.867968] Lustre: Echo OBD driver; http://www.lustre.org/ [12822.038379] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12822.070511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12837.728549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12842.848327] LustreError: 418290:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [12869.536279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12876.444403] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12884.424788] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12891.716867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12891.887566] LustreError: 417626:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eb0f5de4 x1874716542372096/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12892.185127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12892.266832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12896.849510] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12897.253307] LustreError: 418314:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.265890] LustreError: 418314:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12902.371681] LustreError: 418315:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.426313] LustreError: 418314:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.453780] LustreError: 418314:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12906.377721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12906.820271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12912.209224] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12920.972482] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12927.916564] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12937.715941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12937.716941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.737227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12937.743599] Lustre: Skipped 1 previous similar message [12941.665964] Lustre: server umount lustre-OST0000 complete [12947.941632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12947.942887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12947.979896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12947.989111] Lustre: Skipped 2 previous similar messages [12951.484695] Lustre: server umount lustre-MDT0000 complete [12953.056954] LustreError: 419330:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12955.916866] LustreError: 418297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869164 with bad export cookie 15168991264441159810 [12955.924508] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12955.934930] LustreError: 418297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12956.509946] Lustre: server umount lustre-MDT0001 complete [12969.087158] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12972.351283] Key type lgssc unregistered [12972.799045] LNet: 421862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12972.819696] LNetError: 421862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12972.842531] LNet: Removed LNI 192.168.204.154@tcp [12974.030598] Key type .llcrypt unregistered [12974.042650] Key type ._llcrypt unregistered [13000.458711] Key type ._llcrypt registered [13000.462953] Key type .llcrypt registered [13000.585984] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13001.824394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13002.078121] alg: No test for adler32 (adler32-zlib) [13003.231267] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [13003.513459] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13005.208266] Key type lgssc registered [13006.662247] Lustre: Echo OBD driver; http://www.lustre.org/ [13015.394455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13015.415880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13017.182182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13022.218123] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13027.306064] LustreError: 423280:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13027.337601] LustreError: 423280:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13032.418175] LustreError: 423281:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13032.483155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13033.154943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13039.006292] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13048.056663] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13053.950881] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13062.098724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13062.471450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13067.759287] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13076.506215] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13087.134796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13092.321631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13092.330570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13092.346660] Lustre: Skipped 2 previous similar messages [13092.373740] Lustre: Skipped 2 previous similar messages [13093.292586] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [13094.175125] Lustre: server umount lustre-MDT0000 complete [13097.077582] LustreError: 423261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.54@tcp arrived at 1787869305 with bad export cookie 16787038013241820080 [13097.445717] LustreError: 423280:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13097.459114] LustreError: 423280:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13099.273753] LustreError: 423262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869307 with bad export cookie 16787038013241819422 [13099.287932] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13099.299272] LustreError: 423262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13099.492743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13099.509905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13099.529435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13102.562366] LustreError: 423281:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.565673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13102.600619] LustreError: 423281:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13105.612193] Lustre: server umount lustre-OST0000 complete [13109.733958] LustreError: 423275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13109.754919] LustreError: 423275:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [13113.659970] Lustre: server umount lustre-MDT0001 complete [13125.113617] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13128.353987] Key type lgssc unregistered [13128.706349] LNet: 426994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13128.723967] LNetError: 426994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13128.745817] LNet: Removed LNI 192.168.204.154@tcp [13129.705648] Key type .llcrypt unregistered [13129.709776] Key type ._llcrypt unregistered [13157.652162] Key type ._llcrypt registered [13157.653962] Key type .llcrypt registered [13157.737748] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13159.078451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13159.136641] alg: No test for adler32 (adler32-zlib) [13160.260336] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [13160.578051] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13162.306423] Key type lgssc registered [13163.773736] Lustre: Echo OBD driver; http://www.lustre.org/ [13172.793197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13172.804302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13174.477802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13180.242737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13183.976880] LustreError: 428414:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13183.997443] LustreError: 428414:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13189.089636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13189.100143] LustreError: 428415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13189.570546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13194.056447] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13203.398915] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13210.751562] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13220.121396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13220.504929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13227.512133] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13237.002765] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13244.079855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13254.080550] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [13254.630855] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13254.633507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13254.655491] Lustre: Skipped 1 previous similar message [13254.671987] Lustre: Skipped 1 previous similar message [13259.172347] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [13264.287306] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [13264.297429] Lustre: Skipped 2 previous similar messages [13266.400358] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13266.756807] Lustre: server umount lustre-MDT0001 complete [13269.997860] LustreError: 429142:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13270.029901] LustreError: 429142:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13275.108796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13275.113633] LustreError: 428415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13275.135976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13275.167922] Lustre: Skipped 2 previous similar messages [13276.655383] Lustre: server umount lustre-OST0000 complete [13280.231137] LustreError: 429142:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13281.785916] Lustre: server umount lustre-MDT0000 complete [13298.647737] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13302.390723] Key type lgssc unregistered [13302.725203] LNet: 432123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13302.744395] LNetError: 432123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13302.772562] LNet: Removed LNI 192.168.204.154@tcp [13303.898319] Key type .llcrypt unregistered [13303.903426] Key type ._llcrypt unregistered [13333.911646] Key type ._llcrypt registered [13333.914843] Key type .llcrypt registered [13334.111511] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [13352.619159] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13354.359641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13354.414644] alg: No test for adler32 (adler32-zlib) [13355.749769] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [13356.111538] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13357.848194] Key type lgssc registered [13359.736996] Lustre: Echo OBD driver; http://www.lustre.org/ [13410.458543] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13431.424767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13431.477889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13432.902672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13432.949472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13433.106813] Lustre: lustre-MDT0000: new disk, initializing [13433.254423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13433.286310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13439.568949] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13459.007334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13459.225270] Lustre: lustre-OST0000: new disk, initializing [13459.230047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13459.242546] Lustre: Skipped 1 previous similar message [13459.257827] Lustre: 437166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13459.348634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13467.419103] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13467.844087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13467.858360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13467.910442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13480.389853] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13500.692100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13500.820153] Lustre: 436145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13500.917989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13500.982796] Lustre: lustre-MDT0001: new disk, initializing [13501.187705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13501.290099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13501.317332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13507.053805] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13519.101818] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13519.314544] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13524.964370] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13525.277558] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13534.140737] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [13536.247324] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13536.250064] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13536.265245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13537.732493] Lustre: server umount lustre-MDT0001 complete [13546.475863] LustreError: 436155:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13546.504089] LustreError: 436155:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [13551.589273] LustreError: 436156:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13551.612372] LustreError: 436156:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13556.712804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13556.714566] LustreError: 436155:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13556.737575] Lustre: Skipped 1 previous similar message [13556.741738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13556.767185] Lustre: Skipped 1 previous similar message [13561.834152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13561.835332] LustreError: 436156:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13562.602702] Lustre: server umount lustre-OST0000 complete [13581.280615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13581.755899] Lustre: server umount lustre-MDT0000 complete [13598.478584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13602.273971] Key type lgssc unregistered [13602.614691] LNet: 440921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13602.622874] LNetError: 440921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13602.636605] LNet: Removed LNI 192.168.204.154@tcp [13603.638399] Key type .llcrypt unregistered [13603.640126] Key type ._llcrypt unregistered [13655.121835] Key type ._llcrypt registered [13655.126908] Key type .llcrypt registered [13655.254260] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [13655.645342] Key type .llcrypt unregistered [13655.647379] Key type ._llcrypt unregistered [13674.388763] Key type ._llcrypt registered [13674.398402] Key type .llcrypt registered [13674.608198] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [13676.173629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13676.344077] alg: No test for adler32 (adler32-zlib) [13677.682770] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [13677.905386] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [13679.632180] Key type lgssc registered [13681.075460] Lustre: Echo OBD driver; http://www.lustre.org/ [13713.130223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13713.140699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13714.385269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13714.459222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13714.582407] Lustre: lustre-MDT0000: new disk, initializing [13714.666533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13714.691050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13718.538157] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13728.919466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13728.976697] Lustre: 444305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13729.012647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13729.015711] Lustre: Skipped 1 previous similar message [13729.064259] Lustre: lustre-MDT0001: new disk, initializing [13729.132302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13729.166134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13729.174458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13733.435631] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13744.666106] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13751.747660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13757.916465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13758.164560] Lustre: lustre-OST0000: new disk, initializing [13758.168560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13758.174305] Lustre: 446273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13758.313884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13764.394179] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13766.177937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13766.187816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13766.246376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13773.977141] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13785.569796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13785.581448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13785.596302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13786.596891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13786.598669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13788.230709] Lustre: server umount lustre-MDT0000 complete [13791.985071] LustreError: 444295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870000 with bad export cookie 14183236897413979652 [13791.988260] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13791.996073] LustreError: 444295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13792.228318] Lustre: server umount lustre-MDT0001 complete [13806.818381] Lustre: server umount lustre-OST0000 complete [13820.640798] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:33:46 (1787870026) [13829.361493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13829.903073] LustreError: 448304:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13829.920074] LustreError: 448304:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [13829.973466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13835.085320] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13835.247277] LustreError: 448305:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13840.353988] LustreError: 448304:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13844.019206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13844.549365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13850.477373] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13860.741644] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13868.562720] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13890.547405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13890.764989] Lustre: Found index 0 for lustre-OST0000, updating log [13890.962233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13900.217839] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [13902.285125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13910.649885] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13917.667076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13917.677053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13917.688226] Lustre: Skipped 1 previous similar message [13917.703950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13917.711200] Lustre: Skipped 2 previous similar messages [13919.786564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13919.797671] Lustre: Skipped 2 previous similar messages [13923.522483] Lustre: server umount lustre-OST0000 complete [13941.735862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13941.741358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.750000] Lustre: Skipped 1 previous similar message [13941.753190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.761237] Lustre: Skipped 3 previous similar messages [13947.315429] Lustre: server umount lustre-MDT0000 complete [13949.412022] LustreError: 448299:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13949.426845] LustreError: 448299:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13951.252954] LustreError: 448287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870159 with bad export cookie 14183236897413981052 [13951.262443] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13951.266049] LustreError: 448287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13951.559770] Lustre: server umount lustre-MDT0001 complete [13953.377962] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [13964.318878] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:36:09 (1787870169) [14592.452696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14592.962835] LustreError: 454716:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14593.002337] LustreError: 454716:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [14593.076425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14598.115388] LustreError: 454717:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14598.145345] LustreError: 454717:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14599.000659] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14603.233825] LustreError: 454716:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14609.383668] LustreError: 454717:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14609.421729] LustreError: 454717:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14609.702033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14610.073239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14615.134727] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14624.066876] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14630.739339] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14644.640321] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [14651.708655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14651.890255] Lustre: Found index 0 for lustre-OST0000, updating log [14652.153169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14658.266182] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [14665.226092] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14668.202505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14668.212139] Lustre: Skipped 1 previous similar message [14668.222755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14668.231257] Lustre: Skipped 2 previous similar messages [14673.435221] Lustre: server umount lustre-OST0000 complete [14680.233570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14680.255359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14680.276490] Lustre: Skipped 1 previous similar message [14680.301297] Lustre: Skipped 1 previous similar message [14682.361727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14682.387932] Lustre: Skipped 3 previous similar messages [14685.892636] Lustre: server umount lustre-MDT0000 complete [14686.562150] LustreError: 454717:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14686.614578] LustreError: 454717:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14692.229735] LustreError: 454696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870900 with bad export cookie 14183236897413982130 [14692.241395] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14692.251893] LustreError: 454696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14692.896374] Lustre: server umount lustre-MDT0001 complete [14704.033955] LNet: 1 local NIs in recovery (showing 1): 192.168.204.154@tcp [14724.762964] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [14741.503380] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [14780.340696] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:49:46 (1787870986) [14800.806495] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [14804.061142] Key type lgssc unregistered [14804.474786] LNet: 461360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14804.485240] LNetError: 461360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14915.114605] LNet: Removed LNI 192.168.204.154@tcp [14915.956636] Key type .llcrypt unregistered [14915.959571] Key type ._llcrypt unregistered [14934.772583] Key type ._llcrypt registered [14934.774350] Key type .llcrypt registered [14934.921772] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [14935.929724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14936.156811] alg: No test for adler32 (adler32-zlib) [14937.261977] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [14937.480393] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [14939.176238] Key type lgssc registered [14940.309858] Lustre: Echo OBD driver; http://www.lustre.org/ [14945.098858] LNet: Added LNI 192.168.204.154@tcp100 [8/256/0/180] [14945.887920] LNet: Added LNI 192.168.204.154@tcp101 [8/256/0/180] [14946.739677] LNet: Added LNI 192.168.204.154@tcp102 [8/256/0/180] [14947.607331] LNet: Added LNI 192.168.204.154@tcp103 [8/256/0/180] [14948.606458] LNet: Added LNI 192.168.204.154@tcp104 [8/256/0/180] [14949.398283] LNet: Added LNI 192.168.204.154@tcp105 [8/256/0/180] [14950.354164] LNet: Added LNI 192.168.204.154@tcp106 [8/256/0/180] [14951.464922] LNet: Added LNI 192.168.204.154@tcp107 [8/256/0/180] [14952.489227] LNet: Added LNI 192.168.204.154@tcp108 [8/256/0/180] [14953.554263] LNet: Added LNI 192.168.204.154@tcp109 [8/256/0/180] [14954.547300] LNet: Added LNI 192.168.204.154@tcp110 [8/256/0/180] [14955.553749] LNet: Added LNI 192.168.204.154@tcp111 [8/256/0/180] [14956.612465] LNet: Added LNI 192.168.204.154@tcp112 [8/256/0/180] [14957.605133] LNet: Added LNI 192.168.204.154@tcp113 [8/256/0/180] [14958.661589] LNet: Added LNI 192.168.204.154@tcp114 [8/256/0/180] [14959.772370] LNet: Added LNI 192.168.204.154@tcp115 [8/256/0/180] [14960.851937] LNet: Added LNI 192.168.204.154@tcp116 [8/256/0/180] [14961.640396] LNet: Added LNI 192.168.204.154@tcp117 [8/256/0/180] [14962.526994] LNet: Added LNI 192.168.204.154@tcp118 [8/256/0/180] [14963.524674] LNet: Added LNI 192.168.204.154@tcp119 [8/256/0/180] [14964.348553] LNet: Added LNI 192.168.204.154@tcp120 [8/256/0/180] [14965.276639] LNet: Added LNI 192.168.204.154@tcp121 [8/256/0/180] [14966.133305] LNet: Added LNI 192.168.204.154@tcp122 [8/256/0/180] [14967.148786] LNet: Added LNI 192.168.204.154@tcp123 [8/256/0/180] [14968.127724] LNet: Added LNI 192.168.204.154@tcp124 [8/256/0/180] [14969.252459] LNet: Added LNI 192.168.204.154@tcp125 [8/256/0/180] [14970.360847] LNet: Added LNI 192.168.204.154@tcp126 [8/256/0/180] [14971.638847] LNet: Added LNI 192.168.204.154@tcp127 [8/256/0/180] [14972.760941] LNet: Added LNI 192.168.204.154@tcp128 [8/256/0/180] [14973.884750] LNet: Added LNI 192.168.204.154@tcp129 [8/256/0/180] [14974.924640] LNet: Added LNI 192.168.204.154@tcp130 [8/256/0/180] [14976.023372] LNet: Added LNI 192.168.204.154@tcp131 [8/256/0/180] [14976.984916] LNet: Added LNI 192.168.204.154@tcp132 [8/256/0/180] [14977.972546] LNet: Added LNI 192.168.204.154@tcp133 [8/256/0/180] [14979.005364] LNet: Added LNI 192.168.204.154@tcp134 [8/256/0/180] [14980.032738] LNet: Added LNI 192.168.204.154@tcp135 [8/256/0/180] [14988.353214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14988.394948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14989.493401] Lustre: 464409:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [14989.622623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14989.641849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14989.738159] Lustre: lustre-MDT0000: new disk, initializing [14989.836735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14989.855878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14994.235316] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15006.140843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15006.318264] Lustre: 464426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15006.372364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15006.376568] Lustre: Skipped 1 previous similar message [15006.500735] Lustre: lustre-MDT0001: new disk, initializing [15006.551664] Lustre: 465224:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [15006.563567] Lustre: 465224:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15006.741090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15006.818078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15006.837031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15012.530565] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15014.519790] Lustre: 465558:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [15014.542635] Lustre: 465558:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15024.145455] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15031.944617] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15040.475240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15040.831242] Lustre: lustre-OST0000: new disk, initializing [15040.837591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15040.848663] Lustre: 466398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15041.041141] Lustre: 466403:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [15041.051162] Lustre: 466403:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15041.112808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15048.392577] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15050.306600] Lustre: 466841:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.154@tcp: can't add NID 192.168.204.154@tcp130: rc = -75 [15050.323563] Lustre: 466841:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15050.437166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15050.440919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15050.548958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15059.263269] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15071.209798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15071.212456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15071.236231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15071.253607] Lustre: Skipped 1 previous similar message [15073.947568] Lustre: server umount lustre-OST0000 complete [15078.370610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15078.383400] LustreError: Skipped 1 previous similar message [15078.386499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15078.397415] Lustre: Skipped 1 previous similar message [15078.405864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15081.443675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15081.445381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15084.726440] Lustre: server umount lustre-MDT0000 complete [15088.510960] LustreError: 465222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871296 with bad export cookie 15075372120021049724 [15088.519305] LustreError: 465222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15088.520602] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15089.033563] Lustre: server umount lustre-MDT0001 complete [15100.161756] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15103.480472] Key type lgssc unregistered [15103.938935] LNet: 468314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15103.973987] LNetError: 468314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15103.978985] LNet: Removed LNI 192.168.204.154@tcp [15104.994154] LNet: Removed LNI 192.168.204.154@tcp100 [15105.005281] LNetError: 468314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15105.024773] LNetError: 468314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15105.041841] LNet: Removed LNI 192.168.204.154@tcp101 [15105.050946] LNet: Removed LNI 192.168.204.154@tcp102 [15105.060173] LNet: Removed LNI 192.168.204.154@tcp103 [15105.068417] LNet: Removed LNI 192.168.204.154@tcp104 [15105.072506] LNet: Removed LNI 192.168.204.154@tcp105 [15105.085180] LNet: Removed LNI 192.168.204.154@tcp106 [15105.087378] LNet: Removed LNI 192.168.204.154@tcp107 [15105.098892] LNet: Removed LNI 192.168.204.154@tcp108 [15105.106447] LNet: Removed LNI 192.168.204.154@tcp109 [15105.113476] LNet: Removed LNI 192.168.204.154@tcp110 [15105.120278] LNet: Removed LNI 192.168.204.154@tcp111 [15105.127959] LNet: Removed LNI 192.168.204.154@tcp112 [15105.141499] LNet: Removed LNI 192.168.204.154@tcp113 [15105.145383] LNet: Removed LNI 192.168.204.154@tcp114 [15105.160237] LNet: Removed LNI 192.168.204.154@tcp115 [15105.173763] LNet: Removed LNI 192.168.204.154@tcp116 [15105.179433] LNet: Removed LNI 192.168.204.154@tcp117 [15105.187976] LNet: Removed LNI 192.168.204.154@tcp118 [15105.203059] LNet: Removed LNI 192.168.204.154@tcp119 [15105.209513] LNet: Removed LNI 192.168.204.154@tcp120 [15105.220993] LNet: Removed LNI 192.168.204.154@tcp121 [15105.231654] LNet: Removed LNI 192.168.204.154@tcp122 [15105.237489] LNet: Removed LNI 192.168.204.154@tcp123 [15105.240273] LNet: Removed LNI 192.168.204.154@tcp124 [15105.247117] LNet: Removed LNI 192.168.204.154@tcp125 [15105.252122] LNet: Removed LNI 192.168.204.154@tcp126 [15105.255799] LNet: Removed LNI 192.168.204.154@tcp127 [15105.260036] LNet: Removed LNI 192.168.204.154@tcp128 [15105.263492] LNet: Removed LNI 192.168.204.154@tcp129 [15105.267355] LNet: Removed LNI 192.168.204.154@tcp130 [15105.276484] LNet: Removed LNI 192.168.204.154@tcp131 [15105.278444] LNet: Removed LNI 192.168.204.154@tcp132 [15105.288343] LNet: Removed LNI 192.168.204.154@tcp133 [15105.295931] LNet: Removed LNI 192.168.204.154@tcp134 [15105.316071] LNet: Removed LNI 192.168.204.154@tcp135 [15106.191127] Key type .llcrypt unregistered [15106.196131] Key type ._llcrypt unregistered [15132.314501] Key type ._llcrypt registered [15132.317881] Key type .llcrypt registered [15132.466565] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [15148.734838] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15149.898233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15149.911565] alg: No test for adler32 (adler32-zlib) [15150.997288] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [15151.232996] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15152.928255] Key type lgssc registered [15154.409782] Lustre: Echo OBD driver; http://www.lustre.org/ [15190.536995] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:56:36 (1787871396) [15210.510283] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15213.332677] Key type lgssc unregistered [15213.645715] LNet: 472401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15213.664526] LNetError: 472401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15213.683641] LNet: Removed LNI 192.168.204.154@tcp [15214.485572] Key type .llcrypt unregistered [15214.494412] Key type ._llcrypt unregistered [15231.036690] Key type ._llcrypt registered [15231.043513] Key type .llcrypt registered [15231.232398] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15232.449188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15232.501232] alg: No test for adler32 (adler32-zlib) [15233.738898] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [15234.088346] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15235.800327] Key type lgssc registered [15237.076085] Lustre: Echo OBD driver; http://www.lustre.org/ [15247.543508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15247.580300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15248.888236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15248.931179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15249.046873] Lustre: lustre-MDT0000: new disk, initializing [15249.203360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15249.230759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15255.335195] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15267.957618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15268.106047] Lustre: 473616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15268.166618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15268.174427] Lustre: Skipped 1 previous similar message [15268.287858] Lustre: lustre-MDT0001: new disk, initializing [15268.464726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15268.533541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15268.547158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15273.787232] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15284.104634] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15290.419924] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15297.932972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15298.174254] Lustre: lustre-OST0000: new disk, initializing [15298.177519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15298.186339] Lustre: 475587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15298.273754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15304.944643] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15305.793117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15305.803314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15305.906417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15313.995367] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15317.870779] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing lnet_if_list [15323.676774] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15323.722585] LNet: Added LNI 192.168.204.154@tcp10 [8/256/0/180] [15329.127311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15329.163278] LNet: Added LNI 192.168.204.154@tcp11 [8/256/0/180] [15333.720712] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15333.754722] LNet: Added LNI 192.168.204.154@tcp12 [8/256/0/180] [15338.766379] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15338.791360] LNet: Added LNI 192.168.204.154@tcp13 [8/256/0/180] [15344.881246] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15344.907069] LNet: Added LNI 192.168.204.154@tcp14 [8/256/0/180] [15351.929985] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15352.005782] LNet: Added LNI 192.168.204.154@tcp15 [8/256/0/180] [15358.514955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15358.539790] LNet: Added LNI 192.168.204.154@tcp16 [8/256/0/180] [15365.245472] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15365.282019] LNet: Added LNI 192.168.204.154@tcp17 [8/256/0/180] [15371.637331] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15371.679440] LNet: Added LNI 192.168.204.154@tcp18 [8/256/0/180] [15378.066583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15378.112544] LNet: Added LNI 192.168.204.154@tcp19 [8/256/0/180] [15384.160713] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15384.230213] LNet: Added LNI 192.168.204.154@tcp20 [8/256/0/180] [15390.278679] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15390.313490] LNet: Added LNI 192.168.204.154@tcp21 [8/256/0/180] [15396.424169] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15396.460132] LNet: Added LNI 192.168.204.154@tcp22 [8/256/0/180] [15403.256693] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15403.287985] LNet: Added LNI 192.168.204.154@tcp23 [8/256/0/180] [15409.599306] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15409.636020] LNet: Added LNI 192.168.204.154@tcp24 [8/256/0/180] [15416.394192] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15416.455284] LNet: Added LNI 192.168.204.154@tcp25 [8/256/0/180] [15423.693751] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15423.729688] LNet: Added LNI 192.168.204.154@tcp26 [8/256/0/180] [15430.063726] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15430.104897] LNet: Added LNI 192.168.204.154@tcp27 [8/256/0/180] [15437.272396] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15437.314859] LNet: Added LNI 192.168.204.154@tcp28 [8/256/0/180] [15446.100536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15446.137571] LNet: Added LNI 192.168.204.154@tcp29 [8/256/0/180] [15453.570457] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15453.631214] LNet: Added LNI 192.168.204.154@tcp30 [8/256/0/180] [15461.710398] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15461.745826] LNet: Added LNI 192.168.204.154@tcp31 [8/256/0/180] [15469.769890] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15469.824374] LNet: Added LNI 192.168.204.154@tcp32 [8/256/0/180] [15477.448509] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15477.483694] LNet: Added LNI 192.168.204.154@tcp33 [8/256/0/180] [15485.400103] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15485.452977] LNet: Added LNI 192.168.204.154@tcp34 [8/256/0/180] [15494.258900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15494.295087] LNet: Added LNI 192.168.204.154@tcp35 [8/256/0/180] [15502.825793] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15502.854331] LNet: Added LNI 192.168.204.154@tcp36 [8/256/0/180] [15511.213819] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15511.243549] LNet: Added LNI 192.168.204.154@tcp37 [8/256/0/180] [15519.715313] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15519.771043] LNet: Added LNI 192.168.204.154@tcp38 [8/256/0/180] [15528.911222] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15528.960562] LNet: Added LNI 192.168.204.154@tcp39 [8/256/0/180] [15538.228426] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15538.297100] LNet: Added LNI 192.168.204.154@tcp40 [8/256/0/180] [15546.476732] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15546.512264] LNet: Added LNI 192.168.204.154@tcp41 [8/256/0/180] [15555.111576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15555.162762] LNet: Added LNI 192.168.204.154@tcp42 [8/256/0/180] [15564.045842] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15564.068681] LNet: Added LNI 192.168.204.154@tcp43 [8/256/0/180] [15573.165607] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15573.191857] LNet: Added LNI 192.168.204.154@tcp44 [8/256/0/180] [15583.831433] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15583.893416] LNet: Added LNI 192.168.204.154@tcp45 [8/256/0/180] [15595.325670] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15595.392691] LNet: Added LNI 192.168.204.154@tcp46 [8/256/0/180] [15605.421803] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15605.467173] LNet: Added LNI 192.168.204.154@tcp47 [8/256/0/180] [15614.931657] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15614.957917] LNet: Added LNI 192.168.204.154@tcp48 [8/256/0/180] [15624.875444] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15624.902113] LNet: Added LNI 192.168.204.154@tcp49 [8/256/0/180] [15634.552492] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15634.615415] LNet: Added LNI 192.168.204.154@tcp50 [8/256/0/180] [15657.443761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15657.465526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15657.491148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15659.491734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15659.495874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15659.518981] Lustre: Skipped 2 previous similar messages [15661.784382] Lustre: server umount lustre-MDT0000 complete [15666.367459] LustreError: 473607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871874 with bad export cookie 9705126910150642995 [15666.376975] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15666.379052] LustreError: 473607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15666.780542] Lustre: server umount lustre-MDT0001 complete [15685.089156] Lustre: 472885:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871877/real 1787871877] req@0000000070257d65 x1874719085353472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15685.112499] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15685.127546] Lustre: Skipped 1 previous similar message [15686.880450] Lustre: 472886:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871879/real 1787871879] req@000000007a6e94c9 x1874719085353600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15687.247406] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15690.209901] Lustre: 472888:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871882/real 1787871882] req@000000008aedee1f x1874719085353728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15692.258405] Lustre: 472887:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871884/real 1787871884] req@000000008fffa660 x1874719085353984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15697.112031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15697.376216] Lustre: 472885:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871889/real 1787871889] req@00000000795586ea x1874719085354112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787871905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15711.778265] LustreError: 472884:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cade4ba4 x1874719085354880/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15712.087441] LustreError: 475580:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15712.108772] LustreError: 475580:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [15712.134072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15714.209368] LustreError: 475581:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15714.213620] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15715.234485] LustreError: 475580:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15716.431640] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15720.418767] LustreError: 475581:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15725.552637] LustreError: 475580:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15726.265349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15726.755069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15730.865922] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15731.496312] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15739.609555] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15746.018591] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15772.135028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15772.144371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15772.170561] Lustre: Skipped 1 previous similar message [15776.261270] Lustre: server umount lustre-OST0000 complete [15782.369815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15782.371323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15782.380799] Lustre: Skipped 1 previous similar message [15782.395562] Lustre: Skipped 2 previous similar messages [15787.494742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15787.718254] Lustre: server umount lustre-MDT0000 complete [15792.198861] LustreError: 473607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872000 with bad export cookie 9705126910150645599 [15792.207296] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15792.210275] LustreError: 473607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15792.695661] Lustre: server umount lustre-MDT0001 complete [15804.984899] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [15808.149224] Key type lgssc unregistered [15808.545284] LNet: 502648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15808.555378] LNetError: 502648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15808.564224] LNet: Removed LNI 192.168.204.154@tcp [15808.566256] LNet: Removed LNI 192.168.204.154@tcp10 [15808.575715] LNet: Removed LNI 192.168.204.154@tcp11 [15808.579723] LNet: Removed LNI 192.168.204.154@tcp12 [15808.590099] LNet: Removed LNI 192.168.204.154@tcp13 [15808.597789] LNet: Removed LNI 192.168.204.154@tcp14 [15808.604737] LNet: Removed LNI 192.168.204.154@tcp15 [15808.616374] LNet: Removed LNI 192.168.204.154@tcp16 [15808.629206] LNet: Removed LNI 192.168.204.154@tcp17 [15808.637648] LNet: Removed LNI 192.168.204.154@tcp18 [15808.644755] LNet: Removed LNI 192.168.204.154@tcp19 [15808.653361] LNet: Removed LNI 192.168.204.154@tcp20 [15808.661827] LNet: Removed LNI 192.168.204.154@tcp21 [15808.664926] LNet: Removed LNI 192.168.204.154@tcp22 [15808.669652] LNet: Removed LNI 192.168.204.154@tcp23 [15808.675220] LNet: Removed LNI 192.168.204.154@tcp24 [15808.680969] LNet: Removed LNI 192.168.204.154@tcp25 [15808.687774] LNet: Removed LNI 192.168.204.154@tcp26 [15808.689941] LNet: Removed LNI 192.168.204.154@tcp27 [15808.694826] LNet: Removed LNI 192.168.204.154@tcp28 [15808.699506] LNet: Removed LNI 192.168.204.154@tcp29 [15808.703275] LNet: Removed LNI 192.168.204.154@tcp30 [15808.705514] LNet: Removed LNI 192.168.204.154@tcp31 [15808.709293] LNet: Removed LNI 192.168.204.154@tcp32 [15808.713632] LNet: Removed LNI 192.168.204.154@tcp33 [15808.717728] LNet: Removed LNI 192.168.204.154@tcp34 [15808.721352] LNet: Removed LNI 192.168.204.154@tcp35 [15808.725281] LNet: Removed LNI 192.168.204.154@tcp36 [15808.731885] LNet: Removed LNI 192.168.204.154@tcp37 [15808.740229] LNet: Removed LNI 192.168.204.154@tcp38 [15808.747387] LNet: Removed LNI 192.168.204.154@tcp39 [15808.751360] LNet: Removed LNI 192.168.204.154@tcp40 [15808.754710] LNet: Removed LNI 192.168.204.154@tcp41 [15808.762253] LNet: Removed LNI 192.168.204.154@tcp42 [15808.765690] LNet: Removed LNI 192.168.204.154@tcp43 [15808.768696] LNet: Removed LNI 192.168.204.154@tcp44 [15808.770921] LNet: Removed LNI 192.168.204.154@tcp45 [15808.773955] LNet: Removed LNI 192.168.204.154@tcp46 [15808.779858] LNet: Removed LNI 192.168.204.154@tcp47 [15808.783335] LNet: Removed LNI 192.168.204.154@tcp48 [15808.786335] LNet: Removed LNI 192.168.204.154@tcp49 [15808.800644] LNet: Removed LNI 192.168.204.154@tcp50 [15809.704349] Key type .llcrypt unregistered [15809.706710] Key type ._llcrypt unregistered [15835.338520] Key type ._llcrypt registered [15835.340413] Key type .llcrypt registered [15835.488193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [15854.592441] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [15856.275143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15856.372910] alg: No test for adler32 (adler32-zlib) [15857.639516] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [15857.977510] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [15859.696143] Key type lgssc registered [15861.113346] Lustre: Echo OBD driver; http://www.lustre.org/ [15907.396569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15907.434368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15908.830746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15908.861791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15908.944914] Lustre: lustre-MDT0000: new disk, initializing [15909.057850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15909.100919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15914.281920] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15926.049379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15926.207526] Lustre: 505850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15926.262449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15926.267178] Lustre: Skipped 1 previous similar message [15926.336415] Lustre: lustre-MDT0001: new disk, initializing [15926.436704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15926.471074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15926.481160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15932.230749] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15942.981167] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15949.932413] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15957.709784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15957.956284] Lustre: *** cfs_fail_loc=90f, val=0*** [15958.058982] Lustre: lustre-OST0000: new disk, initializing [15958.066963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15958.073990] Lustre: 507869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15958.187089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15965.597717] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [15965.743366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15965.751904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15965.891509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15975.621081] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15986.145594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15986.154768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15986.187525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15986.663928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15986.677244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15986.698108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15990.207580] Lustre: server umount lustre-OST0000 complete [15996.901486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15996.909293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15996.923404] Lustre: Skipped 1 previous similar message [16000.717882] Lustre: server umount lustre-MDT0000 complete [16004.534973] LustreError: 505842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872212 with bad export cookie 7394196280583451582 [16004.540881] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16004.558373] LustreError: 505842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16004.903484] Lustre: server umount lustre-MDT0001 complete [16020.093032] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [16023.908443] Key type lgssc unregistered [16024.274769] LNet: 509780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16024.284787] LNetError: 509780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16024.302103] LNet: Removed LNI 192.168.204.154@tcp [16025.446091] Key type .llcrypt unregistered [16025.450877] Key type ._llcrypt unregistered [16048.155489] Key type ._llcrypt registered [16048.160475] Key type .llcrypt registered [16048.300359] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16050.106924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16050.138168] alg: No test for adler32 (adler32-zlib) [16051.261656] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [16051.628867] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [16053.328133] Key type lgssc registered [16054.517204] Lustre: Echo OBD driver; http://www.lustre.org/ [16062.523498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16062.550450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16064.020408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16068.871250] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16074.220442] LustreError: 511047:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16074.237225] LustreError: 511047:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [16078.237532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16078.690374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16083.789481] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16091.881026] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16097.504151] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16104.638449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16105.019931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16112.058931] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16119.863756] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16134.057061] loop0: detected capacity change from 0 to 204800000 [16134.270549] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16134.287049] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16134.295313] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16140.413976] loop0: detected capacity change from 0 to 204800000 [16140.671669] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16140.686253] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16140.698282] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16146.564395] loop0: detected capacity change from 0 to 204800000 [16152.426533] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16152.569355] Lustre: fs15246-OST0000: new disk, initializing [16152.579365] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16152.587050] Lustre: 514642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16152.645945] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16158.972034] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16168.527334] loop1: detected capacity change from 0 to 204800000 [16173.982725] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16174.107954] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16174.159773] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16174.246350] Lustre: fs15246-MDT0000: new disk, initializing [16174.452011] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16174.472672] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16174.563922] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16175.596821] Lustre: 515764:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000d01efbb x1874719941901568/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787872399 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16179.008884] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16180.661503] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16180.678551] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16180.716361] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16205.806110] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16205.826727] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16208.476655] Lustre: server umount fs15246-MDT0000 complete [16210.920375] LustreError: 511044:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16216.046109] LustreError: 513540:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16219.324844] Lustre: server umount fs15246-OST0000 complete [16230.881472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16230.889505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16230.916704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16236.514595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16236.524605] Lustre: Skipped 3 previous similar messages [16241.634938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16241.648720] Lustre: Skipped 1 previous similar message [16245.216398] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16245.374251] Lustre: server umount lustre-OST0000 complete [16251.889427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16251.894333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16251.905573] Lustre: Skipped 2 previous similar messages [16251.928956] Lustre: Skipped 1 previous similar message [16254.902794] Lustre: server umount lustre-MDT0000 complete [16256.995223] LustreError: 513638:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16257.025659] LustreError: 513638:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [16259.350763] LustreError: 511028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872467 with bad export cookie 14730932336334524823 [16259.368136] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16259.375395] LustreError: 511028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16260.442995] Lustre: server umount lustre-MDT0001 complete [16274.102601] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [16276.905515] Key type lgssc unregistered [16277.190730] LNet: 518866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16277.207485] LNetError: 518866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16277.227890] LNet: Removed LNI 192.168.204.154@tcp [16278.125603] Key type .llcrypt unregistered [16278.130148] Key type ._llcrypt unregistered [16297.297534] Key type ._llcrypt registered [16297.299273] Key type .llcrypt registered [16297.419543] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16298.601662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16298.666292] alg: No test for adler32 (adler32-zlib) [16299.700362] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [16299.964136] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [16301.712244] Key type lgssc registered [16303.055771] Lustre: Echo OBD driver; http://www.lustre.org/ [16343.400397] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [16353.406296] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16392.588471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16392.627864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16393.893140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16393.926248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16394.025313] Lustre: lustre-MDT0000: new disk, initializing [16394.147297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16394.185454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16398.933732] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16410.448770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16410.596910] Lustre: 523330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16410.667899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16410.678335] Lustre: Skipped 1 previous similar message [16410.752702] Lustre: lustre-MDT0001: new disk, initializing [16410.841823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16410.879949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16410.892916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16415.188164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16424.430122] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16430.204399] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16437.571364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16437.941509] Lustre: lustre-OST0000: new disk, initializing [16437.947417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16437.955890] Lustre: 525298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16438.090053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16444.490541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16444.510902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16444.608611] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [16444.656491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16454.330417] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16459.267613] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16459.520936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16464.628830] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16464.881257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16470.499380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16470.505026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16470.514166] LustreError: Skipped 1 previous similar message [16470.517474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16470.528584] Lustre: Skipped 1 previous similar message [16470.547668] Lustre: Skipped 1 previous similar message [16473.394103] Lustre: server umount lustre-OST0000 complete [16477.665651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16477.677539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16477.727631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16480.738930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16480.742536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16480.770337] Lustre: Skipped 1 previous similar message [16483.747355] Lustre: server umount lustre-MDT0000 complete [16487.607079] LustreError: 526839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872695 with bad export cookie 4693545819758199558 [16487.611374] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16487.624417] LustreError: 526839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16488.073137] Lustre: server umount lustre-MDT0001 complete [16514.759515] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [16525.388420] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [16559.586290] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16278 sec ========= 19:19:25 (1787872765) [16561.769419] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:19:27 (1787872767) === [16565.211866] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:19:31 (1787872771) === [16614.882765] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [16618.460827] Key type lgssc unregistered [16618.847712] LNet: 531947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16618.855156] LNetError: 531947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16618.881512] LNet: Removed LNI 192.168.204.154@tcp [16620.075457] Key type .llcrypt unregistered [16620.078691] Key type ._llcrypt unregistered