[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474031201 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007492] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.013056] Yama: becoming mindful. [ 0.014060] SELinux: Initializing. [ 0.015895] *** VALIDATE selinux *** [ 0.023416] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028281] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031097] *** VALIDATE tmpfs *** [ 0.032376] *** VALIDATE proc *** [ 0.034131] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036219] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.041004] Speculative Store Bypass: Vulnerable [ 0.044442] debug: unmapping init [mem 0xffffffffa5459000-0xffffffffa5460fff] [ 0.046186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047655] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053010] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055335] rcu: Hierarchical SRCU implementation. [ 0.057494] smp: Bringing up secondary CPUs ... [ 0.058578] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.063699] smp: Brought up 1 node, 4 CPUs [ 0.065029] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144309] node 0 deferred pages initialised in 77ms [ 0.148113] devtmpfs: initialized [ 0.149290] x86/mm: Memory block size: 128MB [ 0.153000] gcov: version magic: 0x41383552 [ 0.156152] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161334] pinctrl core: initialized pinctrl subsystem [ 0.163146] [ 0.163766] ************************************************************* [ 0.166011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168011] ** ** [ 0.170007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172012] ** ** [ 0.175009] ** This means that this kernel is built to expose internal ** [ 0.176008] ** IOMMU data structures, which may compromise security on ** [ 0.178012] ** your system. ** [ 0.180011] ** ** [ 0.182010] ** If you see this message and you are not debugging the ** [ 0.184014] ** kernel, report this immediately to your vendor! ** [ 0.187012] ** ** [ 0.189012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191013] ************************************************************* [ 0.194665] NET: Registered protocol family 16 [ 0.196328] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206028] cpuidle: using governor menu [ 0.207727] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210470] PCI: Using configuration type 1 for base access [ 0.212093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227123] cryptd: max_cpu_qlen set to 1000 [ 0.229247] ACPI: Added _OSI(Module Device) [ 0.231019] ACPI: Added _OSI(Processor Device) [ 0.234030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242845] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249552] ACPI: Interpreter enabled [ 0.251080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252012] ACPI: Using IOAPIC for interrupt routing [ 0.254099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258288] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268396] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282318] acpiphp: Slot [2] registered [ 0.284124] acpiphp: Slot [5] registered [ 0.286148] acpiphp: Slot [6] registered [ 0.289158] acpiphp: Slot [7] registered [ 0.290160] acpiphp: Slot [8] registered [ 0.292170] acpiphp: Slot [9] registered [ 0.294119] acpiphp: Slot [10] registered [ 0.295084] acpiphp: Slot [3] registered [ 0.297156] acpiphp: Slot [4] registered [ 0.299110] acpiphp: Slot [11] registered [ 0.300127] acpiphp: Slot [12] registered [ 0.302137] acpiphp: Slot [13] registered [ 0.304143] acpiphp: Slot [14] registered [ 0.306111] acpiphp: Slot [15] registered [ 0.307109] acpiphp: Slot [16] registered [ 0.308110] acpiphp: Slot [17] registered [ 0.310094] acpiphp: Slot [18] registered [ 0.312217] acpiphp: Slot [19] registered [ 0.313096] acpiphp: Slot [20] registered [ 0.314114] acpiphp: Slot [21] registered [ 0.316101] acpiphp: Slot [22] registered [ 0.317080] acpiphp: Slot [23] registered [ 0.318078] acpiphp: Slot [24] registered [ 0.319098] acpiphp: Slot [25] registered [ 0.321113] acpiphp: Slot [26] registered [ 0.322083] acpiphp: Slot [27] registered [ 0.323102] acpiphp: Slot [28] registered [ 0.325145] acpiphp: Slot [29] registered [ 0.326084] acpiphp: Slot [30] registered [ 0.327108] acpiphp: Slot [31] registered [ 0.329080] PCI host bridge to bus 0000:00 [ 0.330016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336091] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346449] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350299] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359682] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.365111] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374563] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377789] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382464] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411541] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419059] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456603] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502524] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.519023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.540025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550277] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.556014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.579025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590572] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.598023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.605022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640413] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.671015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682561] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.686459] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689430] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.692439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694303] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702132] iommu: Default domain type: Passthrough [ 0.704438] SCSI subsystem initialized [ 0.706135] ACPI: bus type USB registered [ 0.708097] usbcore: registered new interface driver usbfs [ 0.710081] usbcore: registered new interface driver hub [ 0.712117] usbcore: registered new device driver usb [ 0.714191] pps_core: LinuxPPS API ver. 1 registered [ 0.716010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.719050] PTP clock support registered [ 0.721134] EDAC MC: Ver: 3.0.0 [ 0.723242] PCI: Using ACPI for IRQ routing [ 0.726061] NetLabel: Initializing [ 0.727016] NetLabel: domain hash size = 128 [ 0.729019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.731142] NetLabel: unlabeled traffic allowed by default [ 0.734194] vgaarb: loaded [ 0.736360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744004] clocksource: Switched to clocksource kvm-clock [ 0.852342] VFS: Disk quotas dquot_6.6.0 [ 0.854263] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857134] *** VALIDATE ramfs *** [ 0.858618] *** VALIDATE hugetlbfs *** [ 0.860186] pnp: PnP ACPI init [ 0.862455] pnp: PnP ACPI: found 6 devices [ 0.882552] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.886215] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.888502] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.891168] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893468] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.896159] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.899041] NET: Registered protocol family 2 [ 0.901572] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.907063] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.910452] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.914883] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.917784] TCP: Hash tables configured (established 65536 bind 65536) [ 0.920368] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922802] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.924985] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927323] NET: Registered protocol family 1 [ 0.930241] RPC: Registered named UNIX socket transport module. [ 0.932555] RPC: Registered udp transport module. [ 0.934242] RPC: Registered tcp transport module. [ 0.936089] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.938590] NET: Registered protocol family 44 [ 0.940339] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.942728] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.945020] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.947614] PCI: CLS 0 bytes, default 64 [ 0.949093] Unpacking initramfs... [ 2.321246] debug: unmapping init [mem 0xffff963bfcc54000-0xffff963bfffbffff] [ 2.325346] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.328036] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.331117] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.803330] Initialise system trusted keyrings [ 2.804818] Key type blacklist registered [ 2.806335] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.817787] zbud: loaded [ 2.821115] *** VALIDATE nfs *** [ 2.822213] *** VALIDATE nfs4 *** [ 2.823633] pstore: using deflate compression [ 2.827026] Platform Keyring initialized [ 2.928558] NET: Registered protocol family 38 [ 2.930326] Key type asymmetric registered [ 2.931757] Asymmetric key parser 'x509' registered [ 2.933508] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.936562] io scheduler mq-deadline registered [ 2.938366] io scheduler kyber registered [ 2.940107] io scheduler bfq registered [ 2.941807] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.944625] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.947453] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.950567] ACPI: Power Button [PWRF] [ 2.955320] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.961092] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.972269] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.978617] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.992222] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.020570] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.047664] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.052180] Non-volatile memory driver v1.3 [ 3.054000] Linux agpgart interface v0.103 [ 3.090263] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.094083] vda: detected capacity change from 0 to 75100160 [ 3.110791] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.114455] vdb: detected capacity change from 0 to 1073741824 [ 3.132635] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.135219] vdc: detected capacity change from 0 to 2621440000 [ 3.149951] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.153539] vdd: detected capacity change from 0 to 2621440000 [ 3.168167] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.171420] vde: detected capacity change from 0 to 4294967296 [ 3.186902] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.190263] vdf: detected capacity change from 0 to 4294967296 [ 3.198546] libphy: Fixed MDIO Bus: probed [ 3.206298] usbcore: registered new interface driver usbserial_generic [ 3.208730] usbserial: USB Serial support registered for generic [ 3.211524] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.215905] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.217939] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.220521] mousedev: PS/2 mouse device common for all mice [ 3.223822] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.225547] rtc_cmos 00:05: RTC can wake from S4 [ 3.229107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.231448] rtc_cmos 00:05: registered as rtc0 [ 3.235525] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.239044] intel_pstate: CPU model not supported [ 3.241582] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.246856] hid: raw HID events driver (C) Jiri Kosina [ 3.249318] usbcore: registered new interface driver usbhid [ 3.251714] usbhid: USB HID core driver [ 3.253428] drop_monitor: Initializing network drop monitor service [ 3.256196] Initializing XFRM netlink socket [ 3.258573] NET: Registered protocol family 10 [ 3.262213] Segment Routing with IPv6 [ 3.263804] NET: Registered protocol family 17 [ 3.279475] mpls_gso: MPLS GSO support [ 3.285625] RAS: Correctable Errors collector initialized. [ 3.288040] AVX version of gcm_enc/dec engaged. [ 3.289962] AES CTR mode by8 optimization enabled [ 3.369098] sched_clock: Marking stable (3369041403, 0)->(4252977504, -883936101) [ 3.373585] registered taskstats version 1 [ 3.375858] Loading compiled-in X.509 certificates [ 3.378134] zswap: loaded using pool lzo/zbud [ 3.402243] Key type big_key registered [ 3.415878] Key type encrypted registered [ 3.417950] ima: No TPM chip found, activating TPM-bypass! [ 3.420333] ima: Allocated hash algorithm: sha1 [ 3.422468] ima: No architecture policies found [ 3.424853] evm: Initialising EVM extended attributes: [ 3.426350] evm: security.selinux [ 3.427707] evm: security.ima [ 3.428751] evm: security.capability [ 3.430080] evm: HMAC attrs: 0x1 [ 3.432803] rtc_cmos 00:05: setting system clock to 2026-09-08 01:57:29 UTC (1788832649) [ 3.439315] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.442933] debug: unmapping init [mem 0xffffffffa5182000-0xffffffffa5458fff] [ 3.451089] Write protecting the kernel read-only data: 28672k [ 3.454732] debug: unmapping init [mem 0xffffffffa3803000-0xffffffffa39fffff] [ 3.457670] debug: unmapping init [mem 0xffffffffa4114000-0xffffffffa41fffff] [ 3.494397] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.503544] systemd[1]: Detected virtualization kvm. [ 3.506308] systemd[1]: Detected architecture x86-64. [ 3.509064] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.534435] systemd[1]: No hostname configured. [ 3.536560] systemd[1]: Set hostname to . [ 3.538709] random: systemd: uninitialized urandom read (16 bytes read) [ 3.541863] systemd[1]: Initializing machine ID from random generator. [ 3.603158] random: ln: uninitialized urandom read (6 bytes read) [ 3.691127] random: systemd: uninitialized urandom read (16 bytes read) [ 3.696277] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.700662] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.705673] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.376895] device-mapper: uevent: version 1.0.3 [ 4.379089] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK [ 5.003017] random: fast init done ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.044672] virtio_net virtio0 ens2: renamed from eth0 [ 5.140184] scsi host0: ata_piix [ 5.165576] scsi host1: ata_piix [ 5.170663] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.173423] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.608837] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.939640] random: crng init done [ 9.942291] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.227990] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.316201] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.556430] SELinux: Disabled at runtime. [ 11.620540] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.629280] systemd[1]: Detected virtualization kvm. [ 11.631138] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.099608] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.103364] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.109390] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.112520] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.115021] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.120946] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.127158] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ 12.196581] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.518022] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.865191] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.968853] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.088458] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.104373] EDAC sbridge: Ver: 1.1.2 [ 14.644587] Key type dns_resolver registered [ 14.953173] NFS: Registering the id_resolver key type [ 14.955835] Key type id_resolver registered [ 14.957780] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 40.925545] libcfs: loading out-of-tree module taints kernel. [ 40.949206] Key type ._llcrypt registered [ 40.950874] Key type .llcrypt registered [ 41.008710] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 52.438708] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 54.174726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.201547] alg: No test for adler32 (adler32-zlib) [ 55.759500] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 56.756496] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 58.575205] Key type lgssc registered [ 60.714095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.742699] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.613252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 132.371486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.406600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.683451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.745869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.856328] hrtimer: interrupt took 29317236 ns [ 133.891108] Lustre: lustre-MDT0000: new disk, initializing [ 134.007863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.034068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.769438] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.396626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.546444] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.601469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.607444] Lustre: Skipped 1 previous similar message [ 153.796418] Lustre: lustre-MDT0001: new disk, initializing [ 153.904317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.938089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.953643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.537253] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.766520] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.240963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.603892] Lustre: lustre-OST0000: new disk, initializing [ 175.614089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 175.626053] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.673829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.641457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.654310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.734219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.844242] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.421355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.578498] Lustre: lustre-OST0001: new disk, initializing [ 197.582159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.587566] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.668558] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.826512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.843638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.925433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 204.960223] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.988309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.801609] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.306728] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.281743] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 242.038513] Lustre: DEBUG MARKER: Client: 2.17.57.106 [ 244.900923] Lustre: DEBUG MARKER: MDS: 2.17.57.106 [ 248.114783] Lustre: DEBUG MARKER: OSS: 2.17.57.106 [ 250.130608] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 22:01:34 EDT 2026 [ 269.727995] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 271.393378] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.202363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.214476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.233030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.303509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.307519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.342880] Lustre: Skipped 1 previous similar message [ 293.296546] Lustre: server umount lustre-MDT0000 complete [ 296.972193] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832943 with bad export cookie 6983474298497108767 [ 296.975890] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.001116] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 301.544802] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.545098] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.550179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.562184] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 301.579824] Lustre: Skipped 3 previous similar messages [ 303.395523] Lustre: server umount lustre-MDT0001 complete [ 307.592449] Lustre: server umount lustre-OST0000 complete [ 311.367910] Lustre: server umount lustre-OST0001 complete [ 318.552499] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 328.026688] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 368.163367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 368.414139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 368.450083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 368.553299] Lustre: lustre-MDT0000: new disk, initializing [ 368.674731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.702022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.067886] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 384.671279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.840727] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 384.908156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 384.912206] Lustre: Skipped 1 previous similar message [ 385.008841] Lustre: lustre-MDT0001: new disk, initializing [ 385.107668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 385.138629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 385.166322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.368505] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 397.891130] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 403.758939] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 410.680678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 411.058926] Lustre: lustre-OST0000: new disk, initializing [ 411.062220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 411.069700] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 411.149833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.184661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 412.198443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 412.346446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 417.396509] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 426.893243] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 432.600359] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 432.835585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 437.627821] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 437.850589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 441.322551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.331800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.346126] Lustre: Skipped 1 previous similar message [ 441.364948] Lustre: Skipped 2 previous similar messages [ 446.452259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.457568] Lustre: Skipped 1 previous similar message [ 454.111332] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 454.266496] Lustre: server umount lustre-OST0000 complete [ 461.793689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.802446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.822762] Lustre: Skipped 2 previous similar messages [ 461.823359] Lustre: Skipped 1 previous similar message [ 464.124139] Lustre: server umount lustre-MDT0000 complete [ 466.913613] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.940648] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 467.722463] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833113 with bad export cookie 6983474298497110300 [ 467.727244] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 467.738631] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 468.263783] Lustre: server umount lustre-MDT0001 complete [ 473.087858] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:05:17 (1788833117) [ 475.769324] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 477.566637] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:05:22 (1788833122) [ 484.695608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.305019] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.393105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.468880] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.486722] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 495.584948] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.184102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.333808] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 514.610882] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 521.783532] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 529.463662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.464716] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 545.240898] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 561.645671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 561.653166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.664255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 561.671470] Lustre: Skipped 2 previous similar messages [ 563.816506] Lustre: server umount lustre-OST0000 complete [ 573.950976] Lustre: server umount lustre-MDT0000 complete [ 577.003173] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.046045] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 577.986670] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833224 with bad export cookie 6983474298497111182 [ 577.991465] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.006816] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 578.441967] Lustre: server umount lustre-MDT0001 complete [ 589.837682] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 592.580814] Key type lgssc unregistered [ 593.029818] LNet: 23014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.045771] LNetError: 23014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.085736] LNet: Removed LNI 192.168.203.150@tcp [ 595.169259] Key type .llcrypt unregistered [ 595.172486] Key type ._llcrypt unregistered [ 617.861605] Key type ._llcrypt registered [ 617.864301] Key type .llcrypt registered [ 617.967933] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 618.871883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 619.005983] alg: No test for adler32 (adler32-zlib) [ 620.137495] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 620.359981] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 622.007217] Key type lgssc registered [ 623.101303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.640908] loop: module loaded [ 631.709402] loop0: detected capacity change from 0 to 204800000 [ 631.870458] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.884094] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.116205] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.803215] loop0: detected capacity change from 0 to 204800000 [ 637.957224] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.973235] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.987844] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.942308] loop0: detected capacity change from 0 to 204800000 [ 648.619713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.640733] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.884305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 649.952826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 650.038774] Lustre: lustre-MDT0000: new disk, initializing [ 650.124156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.149249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 654.393534] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 662.989150] loop1: detected capacity change from 0 to 204800000 [ 668.183097] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.529458] Lustre: lustre-OST07c6: new disk, initializing [ 668.535761] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 668.543374] Lustre: Skipped 1 previous similar message [ 668.547747] Lustre: 26201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 668.639474] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 675.375575] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 679.055475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 679.063189] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 679.101183] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 681.528759] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 681.542587] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 681.555303] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 681.565751] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 681.576911] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 681.587839] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 698.492455] Lustre: 26208:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 698.515995] Lustre: 26208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.527070] Lustre: 26208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.540899] Lustre: 26208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 698.550157] Lustre: 26208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.566051] Lustre: 26208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.003780] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 704.014219] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.024822] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.037863] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.051034] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.063677] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.077047] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 704.084276] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.095059] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 704.105743] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.112237] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.118702] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.750106] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 708.765639] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.772910] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 708.778665] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.783893] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.789530] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.794910] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 708.800485] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.805474] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.810305] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.815540] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.820847] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.756526] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 716.789486] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 716.798941] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 716.804255] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.808838] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.815694] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.819973] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 716.829325] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.836201] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.840956] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 716.851718] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.858227] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 726.503220] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 726.511829] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 726.515973] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 726.523511] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 726.535693] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 726.550204] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 726.561711] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 726.577837] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 726.591365] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 726.599935] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 726.616126] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.636255] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.052600] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.065972] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 776.164323] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 776.409847] Lustre: server umount lustre-OST07c6 complete [ 786.149575] Lustre: server umount lustre-MDT0000 complete [ 803.642733] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 806.474192] Key type lgssc unregistered [ 806.744577] LNet: 28756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.753648] LNetError: 28756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.778314] LNet: Removed LNI 192.168.203.150@tcp [ 807.503458] Key type .llcrypt unregistered [ 807.509495] Key type ._llcrypt unregistered [ 829.426470] Key type ._llcrypt registered [ 829.428630] Key type .llcrypt registered [ 829.544650] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 830.864129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.057564] alg: No test for adler32 (adler32-zlib) [ 832.189636] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 832.518966] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 834.207233] Key type lgssc registered [ 835.542867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 844.047893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 844.093618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.685316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.811638] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 855.011938] LustreError: 30024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.031406] LustreError: 30024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 859.991393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.137174] LustreError: 30023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.432128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.187534] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 872.804665] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 878.579779] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.423455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.192518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.309271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 894.772597] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 904.186315] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 926.185932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.188527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 926.216475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.220032] LustreError: Skipped 1 previous similar message [ 927.803707] Lustre: server umount lustre-OST0000 complete [ 932.320645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 932.329585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.344722] Lustre: Skipped 1 previous similar message [ 932.353485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.356740] Lustre: Skipped 1 previous similar message [ 936.416627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.416780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.356204] Lustre: server umount lustre-MDT0000 complete [ 941.541084] LustreError: 30024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.554277] LustreError: 30024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 942.655750] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833588 with bad export cookie 3188610785535113102 [ 942.675216] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 943.065951] Lustre: server umount lustre-MDT0001 complete [ 955.076528] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 957.977947] Key type lgssc unregistered [ 958.321717] LNet: 33761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.332314] LNetError: 33761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.367304] LNet: Removed LNI 192.168.203.150@tcp [ 959.298983] Key type .llcrypt unregistered [ 959.305316] Key type ._llcrypt unregistered [ 979.107946] Key type ._llcrypt registered [ 979.110385] Key type .llcrypt registered [ 979.210635] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 980.496846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 980.667484] alg: No test for adler32 (adler32-zlib) [ 981.763897] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 982.032632] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 983.719242] Key type lgssc registered [ 985.240180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.873503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.886029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.442139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.107940] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1005.546775] LustreError: 35030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.578274] LustreError: 35030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1009.630363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.212194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.690382] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1023.518668] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1030.799133] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1038.923609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.423233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.445155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1046.310695] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1055.323986] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.079587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.081173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.081864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.111814] Lustre: Skipped 1 previous similar message [ 1072.426322] Lustre: server umount lustre-OST0000 complete [ 1076.705571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1076.719206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.733962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.751887] Lustre: Skipped 1 previous similar message [ 1081.313042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.314061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.797703] Lustre: server umount lustre-MDT0000 complete [ 1086.434785] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.457827] LustreError: 35025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1087.151965] LustreError: 35012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833733 with bad export cookie 1950649401177166430 [ 1087.170483] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1087.767408] Lustre: server umount lustre-MDT0001 complete [ 1100.882907] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1104.014963] Key type lgssc unregistered [ 1104.339126] LNet: 38565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.352273] LNetError: 38565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.365276] LNet: Removed LNI 192.168.203.150@tcp [ 1105.305832] Key type .llcrypt unregistered [ 1105.308097] Key type ._llcrypt unregistered [ 1133.033339] Key type ._llcrypt registered [ 1133.035649] Key type .llcrypt registered [ 1133.161466] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1134.633923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.730391] alg: No test for adler32 (adler32-zlib) [ 1135.756498] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1136.016988] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1137.767233] Key type lgssc registered [ 1138.900468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1147.744523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1147.764801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.322166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.421986] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1159.666153] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.701660] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1163.721681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.212444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1169.008626] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1176.865213] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1183.416372] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1191.065984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.471226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.551175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1197.800654] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1206.660964] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1220.040988] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1220.070571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.088791] Lustre: Skipped 1 previous similar message [ 1223.666541] Lustre: server umount lustre-MDT0000 complete [ 1225.142415] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.211869] LustreError: 39961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833874 with bad export cookie 1061117047657556919 [ 1228.223362] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.229155] LustreError: 39961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1230.263715] LustreError: 42240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.279507] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 1230.280105] LustreError: 42240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1230.288024] Lustre: Skipped 3 previous similar messages [ 1230.305319] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.316842] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.317538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1233.345528] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1233.371402] Lustre: Skipped 1 previous similar message [ 1234.740791] Lustre: server umount lustre-MDT0001 complete [ 1250.278975] Lustre: server umount lustre-OST0000 complete [ 1270.168455] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1274.473572] Key type lgssc unregistered [ 1274.961751] LNet: 43839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.972227] LNetError: 43839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.002469] LNet: Removed LNI 192.168.203.150@tcp [ 1276.280722] Key type .llcrypt unregistered [ 1276.283757] Key type ._llcrypt unregistered [ 1305.406532] Key type ._llcrypt registered [ 1305.408550] Key type .llcrypt registered [ 1305.497843] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1306.576156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.659331] alg: No test for adler32 (adler32-zlib) [ 1307.998660] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1308.393229] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1310.208079] Key type lgssc registered [ 1311.722189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1322.105476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1322.156894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.821382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.757628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1333.217811] LustreError: 45252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1333.252746] LustreError: 45252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1338.337373] LustreError: 45251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.978262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.620921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.945681] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1354.665352] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1361.845774] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1370.103320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.614747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.808780] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1386.876689] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1393.205297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1399.246562] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1399.265976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1399.271752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.364073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.369381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.394809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.417107] Lustre: Skipped 2 previous similar messages [ 1405.009739] Lustre: server umount lustre-OST0000 complete [ 1423.841581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.852485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.856990] Lustre: Skipped 2 previous similar messages [ 1427.941994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.952544] Lustre: Skipped 1 previous similar message [ 1429.728687] Lustre: server umount lustre-MDT0000 complete [ 1433.058872] LustreError: 47354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.073210] LustreError: 47354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1433.689556] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834079 with bad export cookie 12596047998485010565 [ 1433.697778] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.699295] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1440.168057] Lustre: server umount lustre-MDT0001 complete [ 1452.165750] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1455.114816] Key type lgssc unregistered [ 1455.489828] LNet: 48962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.501894] LNetError: 48962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.540162] LNet: Removed LNI 192.168.203.150@tcp [ 1456.471922] Key type .llcrypt unregistered [ 1456.476688] Key type ._llcrypt unregistered [ 1486.591232] Key type ._llcrypt registered [ 1486.593839] Key type .llcrypt registered [ 1486.679762] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1487.785394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.819085] alg: No test for adler32 (adler32-zlib) [ 1489.075487] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1489.529504] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1491.216436] Key type lgssc registered [ 1492.438568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1500.206158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1500.232924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.849417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.902902] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1511.396068] LustreError: 50376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1511.419506] LustreError: 50376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1516.164512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.569578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1521.397599] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1529.213775] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1535.590743] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1544.243682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1545.023546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1546.068271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1552.093513] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1560.456473] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1567.352819] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1571.371985] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1583.751859] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0722h23m48s [ 1585.681160] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1588.200936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1588.218119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.248023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.199879] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1589.207217] Lustre: Skipped 2 previous similar messages [ 1590.254953] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1590.266034] Lustre: Skipped 3 previous similar messages [ 1594.042488] Lustre: server umount lustre-MDT0000 complete [ 1595.310712] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.510350] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1601.458607] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.484367] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1604.238868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.444775] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1604.689589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.774191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1605.746885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1609.537461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1613.386675] Lustre: DEBUG MARKER: Wait for df (36666) ... [ 1614.793548] Lustre: DEBUG MARKER: done [ 1619.938150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1619.949491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.965212] Lustre: Skipped 2 previous similar messages [ 1619.973679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.485599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.378817] Lustre: server umount lustre-OST0000 complete [ 1631.716587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.716629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.722297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1631.722306] Lustre: Skipped 2 previous similar messages [ 1631.733159] Lustre: Skipped 1 previous similar message [ 1635.395156] Lustre: server umount lustre-MDT0000 complete [ 1638.880100] LustreError: 50376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.896475] LustreError: 50376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1639.226392] LustreError: 50355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834285 with bad export cookie 6452613564880105411 [ 1639.230043] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.243710] LustreError: 50355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1639.575772] Lustre: server umount lustre-MDT0001 complete [ 1650.829940] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1653.927975] Key type lgssc unregistered [ 1654.458586] LNet: 55298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.466286] LNetError: 55298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.494420] LNet: Removed LNI 192.168.203.150@tcp [ 1655.474171] Key type .llcrypt unregistered [ 1655.483141] Key type ._llcrypt unregistered [ 1700.191287] Key type ._llcrypt registered [ 1700.194935] Key type .llcrypt registered [ 1700.340287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1701.654103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1701.716919] alg: No test for adler32 (adler32-zlib) [ 1702.921761] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1703.248577] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1704.975389] Key type lgssc registered [ 1706.132563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1714.064795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1714.099331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.487130] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1715.521793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1715.678917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.890680] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1728.963111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.105391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1729.131808] Lustre: 57501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1729.248590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1729.373716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1733.843399] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1741.533472] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1747.985370] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1756.212391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.477120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1756.768103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.911172] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1764.026418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1772.092695] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1779.460617] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0722h27m04s [ 1781.236899] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1788.569849] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1791.411865] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1791.441888] LustreError: 57510:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1811.951854] Lustre: lustre-MDT0000: Client feb83d52-15ad-48c9-b265-25692999907c (at 192.168.203.50@tcp) reconnecting [ 1831.442679] LustreError: 57510:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1834.688463] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1838.014736] Lustre: DEBUG MARKER: done [ 1846.245098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1846.248159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.256389] LustreError: Skipped 1 previous similar message [ 1846.257205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.300107] Lustre: Skipped 1 previous similar message [ 1851.383875] Lustre: server umount lustre-OST0000 complete [ 1856.480829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.491708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.511729] Lustre: Skipped 1 previous similar message [ 1856.533630] Lustre: Skipped 2 previous similar messages [ 1862.647751] Lustre: server umount lustre-MDT0000 complete [ 1866.397205] LustreError: 59363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834512 with bad export cookie 17731948913681566363 [ 1866.406099] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1866.839740] Lustre: server umount lustre-MDT0001 complete [ 1877.713742] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1880.693872] Key type lgssc unregistered [ 1881.033616] LNet: 61834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.040425] LNetError: 61834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.052736] LNet: Removed LNI 192.168.203.150@tcp [ 1881.989296] Key type .llcrypt unregistered [ 1881.992225] Key type ._llcrypt unregistered [ 1931.930540] Key type ._llcrypt registered [ 1931.933043] Key type .llcrypt registered [ 1932.020576] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1933.086351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.144542] alg: No test for adler32 (adler32-zlib) [ 1934.256414] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 1934.537693] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1936.215195] Key type lgssc registered [ 1937.799720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.209870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1942.239331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.620780] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1943.640955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1943.852041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.900359] Lustre: Failing over lustre-MDT0000 [ 1945.237087] Lustre: server umount lustre-MDT0000 complete [ 1953.128877] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:29:58 (1788834598) [ 1961.471037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.844182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.109476] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1979.698795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.856085] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1979.973201] Lustre: 64631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1980.587183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1986.171786] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1994.763480] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.319811] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2007.056534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.278990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2007.457559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.881404] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2016.435942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2022.211787] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2027.439715] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2036.706381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.726666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.749682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.238710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.256889] Lustre: Skipped 1 previous similar message [ 2037.261611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.272616] Lustre: Skipped 2 previous similar messages [ 2040.356536] Lustre: server umount lustre-MDT0000 complete [ 2044.442214] LustreError: 64621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834690 with bad export cookie 2513680750394124878 [ 2044.450068] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.456639] LustreError: 64621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2044.882881] Lustre: server umount lustre-MDT0001 complete [ 2048.422290] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2053.201450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2063.177778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.839453] Lustre: 63553:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834693/real 1788834693] req@00000000c8810ef3 x1875729008599424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.887270] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.631178] Lustre: 63554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834695/real 1788834695] req@0000000056e594c0 x1875729008599552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2067.935290] Lustre: 63551:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834698/real 1788834698] req@000000001014cb38 x1875729008599680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.023808] LustreError: 63550:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fd912f78 x1875729008600448/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2069.299038] LustreError: 66480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.319054] LustreError: 66480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2069.433703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.548599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2070.367204] Lustre: 63551:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834700/real 1788834700] req@00000000b9430194 x1875729008599936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.462098] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.588359] LustreError: 68401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.634562] LustreError: 68401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2075.619921] Lustre: 63553:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834705/real 1788834705] req@000000001cb508b7 x1875729008600064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.977414] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2078.692161] LustreError: 68404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.709446] LustreError: 68404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2083.811427] LustreError: 66480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.860152] LustreError: 66480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2085.717792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.143223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.506203] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2091.728128] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2092.531515] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2100.512520] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2106.185514] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.074985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.096382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.117630] Lustre: Skipped 2 previous similar messages [ 2116.129549] Lustre: Skipped 1 previous similar message [ 2119.511891] Lustre: server umount lustre-MDT0000 complete [ 2121.185670] LustreError: 68404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.205756] LustreError: 68404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2124.526341] LustreError: 64621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834770 with bad export cookie 2513680750394127153 [ 2124.544448] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.548405] LustreError: 64621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2125.059451] Lustre: server umount lustre-MDT0001 complete [ 2134.184626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2142.688062] Lustre: 63554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834772/real 1788834772] req@000000008ae87c76 x1875729008629888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788834788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.749577] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.836830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.920197] LustreError: 63550:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c3f76a10 x1875729008630912/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2150.539275] LustreError: 69138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.650759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.778695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2156.250120] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2165.598302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.246532] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2171.755588] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2179.478404] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2185.246664] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2191.842659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.853828] Lustre: Skipped 2 previous similar messages [ 2191.854462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.863249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.864662] Lustre: Skipped 3 previous similar messages [ 2192.092393] Lustre: server umount lustre-MDT0000 complete [ 2196.420610] LustreError: 67343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834842 with bad export cookie 2513680750394128532 [ 2196.421709] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.438337] LustreError: 67343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2196.957165] Lustre: server umount lustre-MDT0001 complete [ 2203.953879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2208.026260] Lustre: DEBUG MARKER: files compared the same [ 2220.825500] Lustre: server umount lustre-OST0000 complete [ 2236.841676] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2239.827490] Key type lgssc unregistered [ 2240.158424] LNet: 74366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.168579] LNetError: 74366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.182593] LNet: Removed LNI 192.168.203.150@tcp [ 2241.092686] Key type .llcrypt unregistered [ 2241.095268] Key type ._llcrypt unregistered [ 2266.093345] Key type ._llcrypt registered [ 2266.095983] Key type .llcrypt registered [ 2266.204170] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2267.438542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.507772] alg: No test for adler32 (adler32-zlib) [ 2268.660689] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2269.013817] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2270.751337] Key type lgssc registered [ 2272.102959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2281.116166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2281.149176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.714367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.403891] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 2292.192351] LustreError: 75786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.208535] LustreError: 75786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2296.798326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.133590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.839311] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 2310.832697] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.605134] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2327.425104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2328.052347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.510820] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug +malloc all [ 2344.265307] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2352.619279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2367.968524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.969650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.976107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.976878] LustreError: Skipped 1 previous similar message [ 2368.008509] Lustre: Skipped 1 previous similar message [ 2369.805677] Lustre: server umount lustre-OST0000 complete [ 2374.112446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.116244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.143537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.146221] Lustre: Skipped 1 previous similar message [ 2378.210769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.212416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.928216] Lustre: server umount lustre-MDT0000 complete [ 2383.330627] LustreError: 75786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.352376] LustreError: 75786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2383.954586] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835030 with bad export cookie 15163183473487578753 [ 2383.957661] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.971080] LustreError: 75765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2384.406563] Lustre: server umount lustre-MDT0001 complete [ 2395.822403] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2398.689445] Key type lgssc unregistered [ 2398.970986] LNet: 79341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.981607] LNetError: 79341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.000897] LNet: Removed LNI 192.168.203.150@tcp [ 2399.775742] Key type .llcrypt unregistered [ 2399.778171] Key type ._llcrypt unregistered [ 2423.976943] Key type ._llcrypt registered [ 2423.980491] Key type .llcrypt registered [ 2424.089073] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2426.028960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2426.044391] alg: No test for adler32 (adler32-zlib) [ 2427.165940] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2427.472097] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2429.199666] Key type lgssc registered [ 2430.681226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2439.698707] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2439.730148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2455.263580] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2460.383911] LustreError: 80728:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 2487.071168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.411963] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2501.391526] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2509.090147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.200039] Lustre: *** cfs_fail_loc=706, val=0*** [ 2509.208577] LustreError: 80063:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000088792d9d x1875729525508352/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2509.547195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.611806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2514.336709] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2514.914045] LustreError: 80753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.933992] LustreError: 80753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2520.033600] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.062816] LustreError: 80753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.081604] LustreError: 80753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2522.996604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.391852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2528.134780] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2536.482990] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2542.517963] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2545.633347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2545.639366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.650537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.219127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.221917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.224881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2551.182549] Lustre: server umount lustre-OST0000 complete [ 2559.458291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.458842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2559.490675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.498628] Lustre: Skipped 2 previous similar messages [ 2561.287632] Lustre: server umount lustre-MDT0000 complete [ 2564.577696] LustreError: 81818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.122286] LustreError: 80734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835211 with bad export cookie 14920913944829511241 [ 2565.128195] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.139920] LustreError: 80734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2565.401671] Lustre: server umount lustre-MDT0001 complete [ 2575.891377] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2578.623525] Key type lgssc unregistered [ 2578.981565] LNet: 84343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.997112] LNetError: 84343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2579.041118] LNet: Removed LNI 192.168.203.150@tcp [ 2579.767423] Key type .llcrypt unregistered [ 2579.769949] Key type ._llcrypt unregistered [ 2602.876819] Key type ._llcrypt registered [ 2602.882823] Key type .llcrypt registered [ 2603.013269] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2603.858657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2603.897940] alg: No test for adler32 (adler32-zlib) [ 2605.502408] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2605.747071] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2607.431234] Key type lgssc registered [ 2608.487672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.302728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.332544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.271522] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2632.282108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.860237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2638.305970] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.348547] LustreError: 86439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2638.809901] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2642.402501] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.419718] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2647.526292] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.553153] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2649.619425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.003593] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.023719] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2650.048209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.202397] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.218338] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2656.441134] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2665.164697] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.184098] Lustre: MGS: Client 44fca7dc-96ed-4add-b483-ce15c1b62dd1 (at 0@lo) reconnecting [ 2665.201886] LustreError: 86453:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2665.433696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.526766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2670.339215] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2688.382870] Lustre: server umount lustre-OST0000 complete [ 2696.165484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.167927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2696.193382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.281923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.293347] Lustre: Skipped 2 previous similar messages [ 2703.368663] Lustre: server umount lustre-MDT0000 complete [ 2706.402566] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.412518] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2717.819383] Lustre: server umount lustre-MDT0001 complete [ 2721.502883] Lustre: server umount MGS complete [ 2732.482165] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:42:57 (1788835377) [ 2760.780252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 2768.947759] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2803.000701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.038681] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2816.426563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.700964] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2816.783637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2816.869283] Lustre: lustre-MDT0001: new disk, initializing [ 2816.926161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2816.949562] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2816.956111] Lustre: 93635:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2816.971304] LustreError: 93635:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2821.438248] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2832.943633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.175160] Lustre: lustre-OST0000: new disk, initializing [ 2833.180651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2833.185332] Lustre: 94401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2833.240271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.110859] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2843.746056] Lustre: 94858:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f17818f0 x1875729712195712/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788835505 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2850.318213] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2857.661818] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.671372] Lustre: MGS: Client b85e8578-a297-4849-aff0-3874b4561266 (at 0@lo) reconnecting [ 2857.715690] Lustre: 95447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2857.747086] LustreError: 93621:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2857.793514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2857.894552] Lustre: lustre-MDT0000: new disk, initializing [ 2858.093669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.129344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2858.148635] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2859.168305] Lustre: 95471:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000047182d79 x1875729712202752/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788835521 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2862.824824] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2864.353039] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2868.335306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2868.343913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2868.432127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2893.795866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.799389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.810872] Lustre: Skipped 2 previous similar messages [ 2898.922944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.931126] Lustre: Skipped 2 previous similar messages [ 2904.038755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.046422] Lustre: Skipped 1 previous similar message [ 2907.103551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2907.337539] Lustre: server umount lustre-OST0000 complete [ 2914.274832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2914.276498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.286710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.293064] Lustre: Skipped 1 previous similar message [ 2914.301732] Lustre: Skipped 1 previous similar message [ 2917.127651] Lustre: server umount lustre-MDT0000 complete [ 2919.402467] LustreError: 93605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.431138] LustreError: 93605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2926.445535] Lustre: server umount lustre-MDT0001 complete [ 2929.725859] Lustre: server umount MGS complete [ 2940.809546] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:46:25 (1788835585) [ 2957.743552] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2960.308832] Key type lgssc unregistered [ 2960.663690] LNet: 97969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.675926] LNetError: 97969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.687867] LNet: Removed LNI 192.168.203.150@tcp [ 2961.597821] Key type .llcrypt unregistered [ 2961.600232] Key type ._llcrypt unregistered [ 2979.476465] Key type ._llcrypt registered [ 2979.479245] Key type .llcrypt registered [ 2979.586200] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2980.701731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2980.743290] alg: No test for adler32 (adler32-zlib) [ 2981.810384] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 2982.017363] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2983.671331] Key type lgssc registered [ 2984.937497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2993.232971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2993.253122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.637543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.736504] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3004.897700] LustreError: 99185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.924838] LustreError: 99185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3007.011217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.388291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.032628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3019.096450] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3025.210864] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3033.102182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3033.402833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.550800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3041.065326] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3049.490436] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3063.267985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.270435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.279235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3063.286643] Lustre: Skipped 1 previous similar message [ 3063.302399] Lustre: Skipped 1 previous similar message [ 3068.389293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3068.400196] Lustre: Skipped 1 previous similar message [ 3069.318725] Lustre: server umount lustre-OST0000 complete [ 3073.516488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.531848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3073.540758] Lustre: Skipped 1 previous similar message [ 3073.554445] Lustre: Skipped 1 previous similar message [ 3078.625691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.645368] Lustre: Skipped 1 previous similar message [ 3079.908871] Lustre: server umount lustre-MDT0000 complete [ 3083.745036] LustreError: 99181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.774475] LustreError: 99181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3083.788735] LustreError: 99167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835729 with bad export cookie 17591103938872319811 [ 3083.804250] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.058532] Lustre: server umount lustre-MDT0001 complete [ 3096.491720] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3099.702522] Key type lgssc unregistered [ 3099.970051] LNet: 102741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3099.990636] LNetError: 102741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3100.013592] LNet: Removed LNI 192.168.203.150@tcp [ 3100.975941] Key type .llcrypt unregistered [ 3100.984060] Key type ._llcrypt unregistered [ 3118.181767] Key type ._llcrypt registered [ 3118.185324] Key type .llcrypt registered [ 3118.330688] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3119.177876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3119.277048] alg: No test for adler32 (adler32-zlib) [ 3120.375506] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3120.645772] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3122.343290] Key type lgssc registered [ 3123.826506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3130.816987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3130.822354] LustreError: 103823:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3130.826283] Lustre: Skipped 1 previous similar message [ 3130.841287] LustreError: 103822:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3130.847794] LustreError: 103823:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3130.851161] LustreError: 103823:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3130.859295] LustreError: 103823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3130.871843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.251525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.647583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.877729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3143.827231] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3148.903838] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing lsmod [ 3151.508970] LustreError: 105029:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3151.511810] LustreError: 105028:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3151.524386] LustreError: 105029:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3151.534680] LustreError: 105029:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3151.539873] LustreError: 105029:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3151.547149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.952083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.424996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.432073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.448231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.965030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.966550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.989783] Lustre: Skipped 1 previous similar message [ 3160.848254] Lustre: server umount lustre-MDT0000 complete [ 3162.086181] LustreError: 103856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.103047] LustreError: 103856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3164.058797] LustreError: 103837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835810 with bad export cookie 13719607805899383319 [ 3164.063827] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3164.072459] LustreError: 103837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3164.288957] Lustre: server umount lustre-MDT0001 complete [ 3177.948481] Lustre: server umount lustre-OST0000 complete [ 3185.989915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.463437] LustreError: 106093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.547838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.309057] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3191.778228] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.921179] LustreError: 106093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.574780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.072568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3205.974383] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3213.992429] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3219.760329] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3226.171906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.471938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.602845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3232.356332] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3239.754347] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3247.075632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3247.084961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.092548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.103838] Lustre: Skipped 1 previous similar message [ 3247.109693] Lustre: Skipped 2 previous similar messages [ 3247.644977] Lustre: server umount lustre-OST0000 complete [ 3252.204409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3252.204765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.212981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.212989] Lustre: Skipped 1 previous similar message [ 3252.239434] Lustre: Skipped 1 previous similar message [ 3257.299727] Lustre: server umount lustre-MDT0000 complete [ 3257.323668] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.342371] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3260.724276] LustreError: 106073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835906 with bad export cookie 13719607805899383998 [ 3260.736802] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.741152] LustreError: 106073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3261.278072] Lustre: server umount lustre-MDT0001 complete [ 3271.405573] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3273.892971] Key type lgssc unregistered [ 3274.228890] LNet: 109623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.241151] LNetError: 109623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.302879] LNet: Removed LNI 192.168.203.150@tcp [ 3276.268376] Key type .llcrypt unregistered [ 3276.272958] Key type ._llcrypt unregistered [ 3296.151773] Key type ._llcrypt registered [ 3296.154645] Key type .llcrypt registered [ 3296.243930] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3297.106475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3297.131765] alg: No test for adler32 (adler32-zlib) [ 3298.224851] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3298.465180] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3300.159180] Key type lgssc registered [ 3301.413817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.494523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.511444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.081852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.401987] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3321.312480] LustreError: 111018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.331389] LustreError: 111018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3323.523624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.914971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.074747] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3335.873837] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3341.501392] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3347.941457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.252902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.188535] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3356.468439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3361.457546] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3367.377434] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3370.462644] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3375.072448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.083808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.103486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.123970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.135219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.140583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3380.268034] Lustre: server umount lustre-OST0000 complete [ 3385.312482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3385.324031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.336084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.342665] Lustre: Skipped 1 previous similar message [ 3387.361283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.365751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.784286] Lustre: server umount lustre-MDT0000 complete [ 3392.481027] LustreError: 111019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.495427] LustreError: 111019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3393.195219] LustreError: 111754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836039 with bad export cookie 18243071742331005548 [ 3393.197815] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.203444] LustreError: 111754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3393.466134] Lustre: server umount lustre-MDT0001 complete [ 3403.267896] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3405.690815] Key type lgssc unregistered [ 3405.917836] LNet: 114653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.927942] LNetError: 114653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3405.955864] LNet: Removed LNI 192.168.203.150@tcp [ 3406.625114] Key type .llcrypt unregistered [ 3406.627146] Key type ._llcrypt unregistered [ 3422.968661] Key type ._llcrypt registered [ 3422.974127] Key type .llcrypt registered [ 3423.057920] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3424.138822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3424.329957] alg: No test for adler32 (adler32-zlib) [ 3425.327512] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3425.554351] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3427.241245] Key type lgssc registered [ 3428.423876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3435.767848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3435.806061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.198266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.286763] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3447.777320] LustreError: 115872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3447.792545] LustreError: 115872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3449.334996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.709953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.457867] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3459.463122] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3464.626385] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3470.441738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.798775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.628036] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3482.124768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3482.362723] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3490.276138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3490.283173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.303850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3492.321369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3492.327081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.344574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3496.468361] Lustre: server umount lustre-OST0000 complete [ 3501.024434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.031659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.051645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.054778] Lustre: Skipped 1 previous similar message [ 3507.681229] LustreError: 115867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.708389] LustreError: 115867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3507.927611] Lustre: server umount lustre-MDT0000 complete [ 3511.399165] LustreError: 115850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836157 with bad export cookie 15149738363804264234 [ 3511.410313] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3511.416302] LustreError: 115850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3511.854509] Lustre: server umount lustre-MDT0001 complete [ 3522.391734] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3525.302959] Key type lgssc unregistered [ 3525.649713] LNet: 119404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3525.655234] LNetError: 119404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3525.671753] LNet: Removed LNI 192.168.203.150@tcp [ 3526.571155] Key type .llcrypt unregistered [ 3526.573779] Key type ._llcrypt unregistered [ 3553.245327] Key type ._llcrypt registered [ 3553.252964] Key type .llcrypt registered [ 3553.358671] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3554.217440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3554.342388] alg: No test for adler32 (adler32-zlib) [ 3555.607626] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3555.862618] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3557.551254] Key type lgssc registered [ 3558.789359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3566.933634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3567.000273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.540337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.623993] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3578.853981] LustreError: 120819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.864622] LustreError: 120819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3582.668814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.017674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3586.887164] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3594.341431] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3600.541733] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3607.350948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3607.464057] LustreError: 122639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.726150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.207314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3614.516483] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3622.706349] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3629.748220] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3634.923233] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3634.932177] Lustre: Skipped 2 previous similar messages [ 3638.976955] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3639.026041] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3640.464917] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3640.470122] Lustre: Skipped 2 previous similar messages [ 3645.034987] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3645.040879] Lustre: Skipped 1 previous similar message [ 3650.907481] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3650.914542] Lustre: Skipped 2 previous similar messages [ 3653.690160] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.50@tcp 192.168.0.[2,10]@tcp [ 3659.021595] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.50/24@tcp in log lustre-MDT0000 [ 3659.028723] Lustre: Skipped 2 previous similar messages [ 3660.731332] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.50/24@tcp [ 3660.737232] Lustre: Skipped 1 previous similar message [ 3669.476560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.486918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.503303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3669.997089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3670.005494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.028374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.618432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.632553] Lustre: Skipped 3 previous similar messages [ 3680.740654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.747259] Lustre: Skipped 1 previous similar message [ 3683.295189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3683.455445] Lustre: server umount lustre-OST0000 complete [ 3690.465769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3690.479203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.495941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3693.733806] Lustre: server umount lustre-MDT0000 complete [ 3696.097497] LustreError: 120818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.119180] LustreError: 120818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3697.648822] LustreError: 120799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836343 with bad export cookie 11240357945280441179 [ 3697.655837] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.660356] LustreError: 120799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3697.963482] Lustre: server umount lustre-MDT0001 complete [ 3709.620913] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3712.779364] Key type lgssc unregistered [ 3713.182623] LNet: 125569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3713.193815] LNetError: 125569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3713.232678] LNet: Removed LNI 192.168.203.150@tcp [ 3714.380325] Key type .llcrypt unregistered [ 3714.383305] Key type ._llcrypt unregistered [ 3745.030277] Key type ._llcrypt registered [ 3745.032592] Key type .llcrypt registered [ 3745.119892] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3746.064046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3746.151116] alg: No test for adler32 (adler32-zlib) [ 3747.258827] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3747.557701] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3749.231176] Key type lgssc registered [ 3750.380631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3758.164087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3758.195681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.718965] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3759.751048] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.50/24@tcp [ 3759.761361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.670876] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3769.315860] LustreError: 127125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.327769] LustreError: 127125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3773.156855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.379569] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3773.406032] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.50/24@tcp [ 3773.469297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3778.084671] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3785.970618] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.395496] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3799.156061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.531700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3804.669623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3805.304450] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3812.836881] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3826.153086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3826.160759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.172665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.218829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.223071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.256628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.263977] Lustre: server umount lustre-OST0000 complete [ 3839.458844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.463111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.480862] Lustre: Skipped 1 previous similar message [ 3839.493697] Lustre: Skipped 2 previous similar messages [ 3842.523391] Lustre: server umount lustre-MDT0000 complete [ 3844.577175] LustreError: 128710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3844.596700] LustreError: 128710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3846.714984] LustreError: 127106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836492 with bad export cookie 12729909897353765490 [ 3846.722109] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.737049] LustreError: 127106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3847.228360] Lustre: server umount lustre-MDT0001 complete [ 3858.728439] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3861.238827] Key type lgssc unregistered [ 3861.515918] LNet: 130726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3861.523381] LNetError: 130726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3861.552953] LNet: Removed LNI 192.168.203.150@tcp [ 3862.402329] Key type .llcrypt unregistered [ 3862.405165] Key type ._llcrypt unregistered [ 3888.522741] Key type ._llcrypt registered [ 3888.524436] Key type .llcrypt registered [ 3888.646981] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3904.175599] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3905.481968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3905.678071] alg: No test for adler32 (adler32-zlib) [ 3906.766784] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 3907.038489] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3908.751447] Key type lgssc registered [ 3909.974962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3941.812552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3941.849382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.070947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3943.096352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3943.201615] Lustre: lustre-MDT0000: new disk, initializing [ 3943.325404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.344390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3947.460652] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3956.418914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.555428] Lustre: 134124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3956.632208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3956.638346] Lustre: Skipped 1 previous similar message [ 3956.700787] Lustre: lustre-MDT0001: new disk, initializing [ 3956.760791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.799204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3956.809729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3960.711233] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3969.313224] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3974.601241] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3981.158569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3981.359466] Lustre: lustre-OST0000: new disk, initializing [ 3981.362867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3981.368297] Lustre: 136091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3981.466619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.988642] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3991.080509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3991.088430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3991.185234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3996.085953] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.306881] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4000.472546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4004.721274] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4004.902201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4011.488523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.498105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.518843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4012.008769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4012.011659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.069730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4017.136444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.215551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4021.429341] Lustre: server umount lustre-OST0000 complete [ 4027.369817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4027.374230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4027.388675] Lustre: Skipped 1 previous similar message [ 4027.397350] Lustre: Skipped 2 previous similar messages [ 4032.482780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.489154] Lustre: Skipped 1 previous similar message [ 4039.136908] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4039.348475] Lustre: server umount lustre-MDT0000 complete [ 4042.537474] LustreError: 134114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836688 with bad export cookie 14103826561928593497 [ 4042.543149] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4042.547244] LustreError: 134114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4042.840344] Lustre: server umount lustre-MDT0001 complete [ 4050.902868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.340673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.179532] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4056.549239] LustreError: 138309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.564491] LustreError: 138309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4061.668466] LustreError: 138308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.345965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.642702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4067.969486] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4074.202313] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4079.456311] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4085.095459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.348893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4090.140728] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4096.755396] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.904353] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4101.116820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4104.615639] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4104.748569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4112.659205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.821552] Lustre: lustre-OST0001: new disk, initializing [ 4112.824988] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4112.830814] Lustre: 141678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4112.922929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.571712] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4122.181515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4122.192397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4122.282760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4124.416705] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4124.630684] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4128.982365] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4129.145923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4134.487638] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98829007d000.ost_server_uuid 50 [ 4136.184840] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98829007d000.ost_server_uuid in IDLE state after 0 sec [ 4143.074392] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4143.079495] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.089661] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4143.094179] Lustre: Skipped 3 previous similar messages [ 4149.101369] Lustre: server umount lustre-OST0001 complete [ 4153.328673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.328743] LustreError: 140131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.334182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4153.334190] LustreError: Skipped 1 previous similar message [ 4153.341464] Lustre: Skipped 2 previous similar messages [ 4153.369916] LustreError: 140131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4158.172802] Lustre: server umount lustre-OST0000 complete [ 4163.556142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.578588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.584374] Lustre: Skipped 5 previous similar messages [ 4167.632090] Lustre: server umount lustre-MDT0000 complete [ 4168.675192] LustreError: 138305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.691880] LustreError: 138305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4170.992585] LustreError: 141677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836817 with bad export cookie 14103826561928594379 [ 4171.004242] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4171.010195] LustreError: 141677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4171.358675] Lustre: server umount lustre-MDT0001 complete [ 4186.588174] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4188.389846] Key type lgssc unregistered [ 4188.642308] LNet: 144682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4188.648110] LNetError: 144682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4188.661596] LNet: Removed LNI 192.168.203.150@tcp [ 4189.367831] Key type .llcrypt unregistered [ 4189.371405] Key type ._llcrypt unregistered [ 4226.322660] Key type ._llcrypt registered [ 4226.326948] Key type .llcrypt registered [ 4226.423214] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4239.885658] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4240.932171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4240.956952] alg: No test for adler32 (adler32-zlib) [ 4241.953539] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 4242.186299] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4243.855309] Key type lgssc registered [ 4245.010302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4273.856315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4273.894765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.069144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4275.104530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4275.154357] Lustre: lustre-MDT0000: new disk, initializing [ 4275.228263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.244757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4278.803797] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4287.657117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.748866] Lustre: 148824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4287.803583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4287.808056] Lustre: Skipped 1 previous similar message [ 4287.884888] Lustre: lustre-MDT0001: new disk, initializing [ 4287.995431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4288.041839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4288.055746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4291.568227] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4299.428850] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4304.083456] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4309.538635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.754696] Lustre: lustre-OST0000: new disk, initializing [ 4309.758546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4309.764172] Lustre: 150792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4309.828841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.616911] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4316.196945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4316.203734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4316.295587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4322.359372] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.575135] Lustre: Failing over lustre-OST0000 [ 4327.736395] Lustre: server umount lustre-OST0000 complete [ 4345.893199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.236920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4346.247258] Lustre: Skipped 1 previous similar message [ 4346.274717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.303566] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4346.356262] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4347.304692] Lustre: 147048:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788836977/real 1788836977] req@000000000b8fca4c x1875731428704640/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788836993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4347.322589] Lustre: 147048:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4348.320877] Lustre: 147049:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788836978/real 1788836978] req@0000000018d203db x1875731428704768/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788836994 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4351.492933] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4351.493101] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4351.966377] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4352.543210] Lustre: 147049:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788836982/real 1788836982] req@00000000e5cbbb6e x1875731428705920/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788836998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4355.783890] Lustre: Failing over lustre-MDT0000 [ 4356.108746] Lustre: server umount lustre-MDT0000 complete [ 4356.578662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.592392] Lustre: Skipped 1 previous similar message [ 4357.663241] Lustre: 147050:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788836987/real 1788836987] req@0000000086f8a9ce x1875731428706944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788837003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4357.692225] Lustre: 147050:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4361.654721] LustreError: 150403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.670198] LustreError: 150403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4366.771382] LustreError: 152705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.807561] LustreError: 152705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4371.894416] LustreError: 150403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.909797] LustreError: 150403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4371.935075] Lustre: 147047:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837002/real 1788837002] req@0000000018d203db x1875731428715520/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788837018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4371.983490] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4375.016605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.011352] LustreError: 148830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.036852] LustreError: 148830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4382.188743] LustreError: 148836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.194079] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x96bc74a83d511434 [ 4382.207819] LustreError: 148836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4382.230691] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [ 4382.233653] Lustre: Skipped 1 previous similar message [ 4382.443285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.507769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4383.152956] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4386.967459] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4387.826958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4387.880176] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4387.935373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4397.537069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4397.541927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.556404] Lustre: Skipped 1 previous similar message [ 4397.569614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.885812] Lustre: server umount lustre-OST0000 complete [ 4408.288308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4408.288898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.291225] Lustre: Skipped 1 previous similar message [ 4408.304107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.328886] Lustre: Skipped 3 previous similar messages [ 4410.824800] Lustre: server umount lustre-MDT0000 complete [ 4413.414888] LustreError: 148836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.437542] LustreError: 148836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4415.081969] LustreError: 148816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837061 with bad export cookie 10861684667242845236 [ 4415.084122] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.095598] LustreError: 148816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4415.476468] Lustre: server umount lustre-MDT0001 complete [ 4427.636097] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 4430.502677] Key type lgssc unregistered [ 4430.827590] LNet: 154447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4430.835323] LNetError: 154447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4430.852187] LNet: Removed LNI 192.168.203.150@tcp [ 4431.901257] Key type .llcrypt unregistered [ 4431.903655] Key type ._llcrypt unregistered [ 4468.844558] Key type ._llcrypt registered [ 4468.847584] Key type .llcrypt registered [ 4468.949823] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4470.098118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4470.115567] alg: No test for adler32 (adler32-zlib) [ 4471.134116] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 4471.375303] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 4473.047192] Key type lgssc registered [ 4474.113446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4480.624741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4480.641422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.007117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.708387] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4492.263602] LustreError: 155963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.273577] LustreError: 155963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4494.049304] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.383523] LustreError: 155964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.869414] LustreError: 155992:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4497.896865] LustreError: 155992:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4497.916508] LustreError: 155992:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4498.437154] Lustre: server umount lustre-MDT0000 complete [ 4517.455181] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4527.839641] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 4570.708643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.048643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4571.084952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4571.198692] Lustre: lustre-MDT0000: new disk, initializing [ 4571.321956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.345651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4575.805297] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4586.078318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.235170] Lustre: 159298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4586.318330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4586.321942] Lustre: Skipped 1 previous similar message [ 4586.419443] Lustre: lustre-MDT0001: new disk, initializing [ 4586.503934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4586.546069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4586.557929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4590.783080] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4600.896599] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4606.425289] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4613.151490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.497553] Lustre: lustre-OST0000: new disk, initializing [ 4613.507253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4613.521283] Lustre: 161267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4613.618827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.886922] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4622.434278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4622.443915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4622.520042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4629.160625] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4633.040823] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4634.664511] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:14:39 (1788837279) [ 4903.915741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4903.921102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.931215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4909.536453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4909.547090] Lustre: Skipped 5 previous similar messages [ 4909.863543] Lustre: server umount lustre-MDT0000 complete [ 4914.232261] LustreError: 159290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837560 with bad export cookie 8433719945703295417 [ 4914.236039] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4914.247145] LustreError: 159290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4914.656095] LustreError: 160165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4914.656675] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4914.716113] Lustre: Skipped 2 previous similar messages [ 4914.724393] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4919.775597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4919.775597] LustreError: 159305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.793318] Lustre: server umount lustre-MDT0001 complete [ 4939.240527] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4939.630850] Lustre: server umount lustre-OST0000 complete [ 4950.849085] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 4960.098704] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5002.526920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.935426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5002.978114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5003.160388] Lustre: lustre-MDT0000: new disk, initializing [ 5003.291120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.317870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5008.699739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5022.306487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.435215] Lustre: 165936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5022.574136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5022.580150] Lustre: Skipped 1 previous similar message [ 5022.730531] Lustre: lustre-MDT0001: new disk, initializing [ 5022.895445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5022.966523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5022.991487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5027.847212] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5039.518238] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5046.749735] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5055.862641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.294729] Lustre: lustre-OST0000: new disk, initializing [ 5056.299232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5056.314253] Lustre: 167903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5056.449870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.876692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5063.903260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5064.133040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5064.279430] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5074.340135] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5079.448982] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5079.728899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5085.850477] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5086.133125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5089.761879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5089.769456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.773421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.785825] Lustre: Skipped 1 previous similar message [ 5089.797266] Lustre: Skipped 1 previous similar message [ 5094.895462] LustreError: 167888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.101583] Lustre: server umount lustre-OST0000 complete [ 5100.000413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5100.002495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.010909] LustreError: Skipped 1 previous similar message [ 5100.045461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.063519] Lustre: Skipped 1 previous similar message [ 5106.048138] Lustre: server umount lustre-MDT0000 complete [ 5110.240830] LustreError: 165947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5110.267062] LustreError: 165947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5110.547301] LustreError: 165928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837756 with bad export cookie 8433719945703454240 [ 5110.559168] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.281993] Lustre: server umount lustre-MDT0001 complete [ 5120.916648] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:22:45 (1788837765) [ 5139.910658] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5151.463336] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5192.814815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.150500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5193.207440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5193.339516] Lustre: lustre-MDT0000: new disk, initializing [ 5193.416806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.483891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5198.824720] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5209.121052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.265976] Lustre: 172666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5209.531249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.301311] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5224.073421] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5229.795962] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5236.847361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.237545] Lustre: lustre-OST0000: new disk, initializing [ 5237.240311] Lustre: Skipped 1 previous similar message [ 5237.245832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5237.255925] Lustre: Skipped 2 previous similar messages [ 5237.269736] Lustre: 174643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5243.425439] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5244.001840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5244.021226] Lustre: Skipped 1 previous similar message [ 5244.030666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5244.173838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5253.671202] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5264.870612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5264.895060] Lustre: Skipped 1 previous similar message [ 5264.903107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5264.916247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5264.930329] Lustre: Skipped 3 previous similar messages [ 5268.109733] Lustre: server umount lustre-OST0000 complete [ 5275.104580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.121206] Lustre: Skipped 1 previous similar message [ 5286.367141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5286.587780] Lustre: server umount lustre-MDT0000 complete [ 5290.464604] LustreError: 173834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.495588] LustreError: 173834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5290.590445] LustreError: 172656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837936 with bad export cookie 8433719945703455129 [ 5290.592055] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.608982] LustreError: 172656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5291.066909] Lustre: server umount lustre-MDT0001 complete [ 5300.722587] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:25:45 (1788837945) [ 5319.556848] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 5333.262622] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5376.838117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.101050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5377.141433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5377.260453] Lustre: lustre-MDT0000: new disk, initializing [ 5377.337849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.345083] Lustre: Skipped 2 previous similar messages [ 5377.370859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5382.904914] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5395.993446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.135065] Lustre: 178971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5396.430145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5402.263946] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5412.363799] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5418.754748] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5426.041679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5426.340669] Lustre: 180946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5431.873907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5432.028119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5433.389605] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5443.918510] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5452.776236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.781982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.807596] LustreError: Skipped 1 previous similar message [ 5452.826485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5452.830113] Lustre: Skipped 2 previous similar messages [ 5452.853171] Lustre: Skipped 7 previous similar messages [ 5458.005638] Lustre: server umount lustre-OST0000 complete [ 5463.010800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5477.344741] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5477.677066] Lustre: server umount lustre-MDT0000 complete [ 5482.464833] LustreError: 181765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5482.503872] LustreError: 181765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5482.657531] LustreError: 178961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838128 with bad export cookie 8433719945703456214 [ 5482.660190] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5482.673480] LustreError: 178961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5483.047777] Lustre: server umount lustre-MDT0001 complete [ 5494.380896] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5497.323936] Key type lgssc unregistered [ 5497.632799] LNet: 182805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5497.649747] LNetError: 182805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5497.667977] LNet: Removed LNI 192.168.203.150@tcp [ 5498.636778] Key type .llcrypt unregistered [ 5498.640710] Key type ._llcrypt unregistered [ 5521.801737] Key type ._llcrypt registered [ 5521.806672] Key type .llcrypt registered [ 5521.961412] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5523.106090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5523.187171] alg: No test for adler32 (adler32-zlib) [ 5524.396329] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 5524.762843] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5526.473857] Key type lgssc registered [ 5527.675629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5535.056569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5535.087580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.463514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.824772] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5545.955028] LustreError: 184216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.970924] LustreError: 184216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5548.310201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.615601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5552.402442] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5558.772681] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5564.368293] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5570.639525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5571.053753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.481728] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5581.320443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5584.643259] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5603.810516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.818754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.831296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5606.880426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5606.886979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5606.888980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.942673] Lustre: Skipped 1 previous similar message [ 5609.978613] Lustre: server umount lustre-OST0000 complete [ 5615.079505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5615.092790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.118102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.120727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.122191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.162100] Lustre: Skipped 1 previous similar message [ 5620.517548] Lustre: server umount lustre-MDT0000 complete [ 5622.244806] LustreError: 184944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.278270] LustreError: 184944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5624.778636] LustreError: 184197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838270 with bad export cookie 4652991363613172870 [ 5624.786150] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5624.797676] LustreError: 184197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5625.146938] Lustre: server umount lustre-MDT0001 complete [ 5637.071213] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5639.646061] Key type lgssc unregistered [ 5639.922720] LNet: 187775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5639.931432] LNetError: 187775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5639.954651] LNet: Removed LNI 192.168.203.150@tcp [ 5640.791836] Key type .llcrypt unregistered [ 5640.796849] Key type ._llcrypt unregistered [ 5669.933849] Key type ._llcrypt registered [ 5669.936364] Key type .llcrypt registered [ 5670.056919] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5670.921724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5670.973707] alg: No test for adler32 (adler32-zlib) [ 5672.181030] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 5672.426311] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5674.153680] Key type lgssc registered [ 5675.518964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5683.903373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5683.938494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.488091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.538313] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5694.950042] LustreError: 189189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.964204] LustreError: 189189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5699.021147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.555212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.175543] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5711.827360] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5717.621508] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5724.919544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5725.310478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5730.486516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5731.951968] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5740.497846] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5751.264924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5751.283361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5751.330600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5753.324483] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5755.362298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.370094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.374782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5755.394339] Lustre: Skipped 1 previous similar message [ 5758.411946] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5763.522991] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 5763.530235] Lustre: Skipped 2 previous similar messages [ 5765.087670] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5765.350435] Lustre: server umount lustre-OST0000 complete [ 5773.849861] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff988292632800.ost_server_uuid 50 [ 5775.814030] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5790.184108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.186098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5790.208759] Lustre: Skipped 1 previous similar message [ 5790.231075] Lustre: Skipped 1 previous similar message [ 5795.858041] Lustre: server umount lustre-MDT0000 complete [ 5799.903365] LustreError: 189169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838445 with bad export cookie 16438675871764399658 [ 5799.910661] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5799.913866] LustreError: 189169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5800.295964] Lustre: server umount lustre-MDT0001 complete [ 5809.079637] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:34:14 (1788838454) [ 5815.111750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.638194] LustreError: 193105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.713904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.619317] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5820.897064] LustreError: 193106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.002441] LustreError: 193105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5830.115234] LustreError: 193106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5830.178551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.645887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5835.384947] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5843.177915] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5849.433543] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5857.373612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.695040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.815695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5865.540666] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5875.023454] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5884.218213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5884.689419] Lustre: lustre-OST0001: new disk, initializing [ 5884.695587] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5884.702167] Lustre: 195948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5884.812190] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5891.470617] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5894.805203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5894.814493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5894.984754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5902.269611] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5905.384478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.389157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.395842] Lustre: Skipped 1 previous similar message [ 5905.405813] Lustre: Skipped 3 previous similar messages [ 5911.407095] Lustre: server umount lustre-OST0000 complete [ 5913.034992] LustreError: 194929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.718794] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5917.958531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5922.932716] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5923.222816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5930.978020] LustreError: 196756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.004157] LustreError: 196756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5941.221200] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.227182] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.235170] LustreError: Skipped 1 previous similar message [ 5941.239030] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5941.257763] Lustre: Skipped 1 previous similar message [ 5941.287024] Lustre: Skipped 4 previous similar messages [ 5942.884967] Lustre: server umount lustre-OST0001 complete [ 5952.456842] Lustre: server umount lustre-MDT0000 complete [ 5956.857558] LustreError: 193086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838602 with bad export cookie 16438675871764400729 [ 5956.859611] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5956.883652] LustreError: 193086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5957.789170] Lustre: server umount lustre-MDT0001 complete [ 5986.007806] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:37:10 (1788838630) [ 5992.073400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5992.303460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5992.358068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5992.602398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.903961] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6005.613915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.747710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6005.818452] Lustre: 199388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6010.722771] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6018.532597] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6024.798958] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6032.273123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6032.404701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6039.168745] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6040.699364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6046.206949] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6053.227538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6053.324522] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6060.468691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6063.302801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6068.213847] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6073.837047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.839352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.858910] Lustre: Skipped 3 previous similar messages [ 6073.883903] Lustre: Skipped 5 previous similar messages [ 6078.204364] Lustre: server umount lustre-OST0000 complete [ 6078.972865] LustreError: 201238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.021791] LustreError: 201238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6099.426579] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6099.442056] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.956803] Lustre: server umount lustre-OST0001 complete [ 6110.941641] Lustre: server umount lustre-MDT0000 complete [ 6114.647502] LustreError: 201257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838760 with bad export cookie 16438675871764402136 [ 6114.649062] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6114.661482] LustreError: 201257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6115.013444] Lustre: server umount lustre-MDT0001 complete [ 6137.244469] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:39:42 (1788838782) [ 6151.401133] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 6160.359975] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6197.362884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.502662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6197.536412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6197.604454] Lustre: lustre-MDT0000: new disk, initializing [ 6197.686629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.691786] Lustre: Skipped 3 previous similar messages [ 6197.706681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6201.860602] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6211.462513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.570959] Lustre: 207478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6211.606798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6211.610380] Lustre: Skipped 1 previous similar message [ 6211.661615] Lustre: lustre-MDT0001: new disk, initializing [ 6211.778194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6211.796744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6216.470912] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6225.217305] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6231.366515] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6237.766095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.946639] Lustre: lustre-OST0000: new disk, initializing [ 6237.949757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6237.954621] Lustre: 209451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6244.237788] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6247.545143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6247.565484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6247.720194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6254.555760] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6259.809933] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6260.122643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6264.752330] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6265.123695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6268.385935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.389162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6268.401414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6268.401423] Lustre: Skipped 7 previous similar messages [ 6268.411854] Lustre: Skipped 4 previous similar messages [ 6268.443736] LustreError: Skipped 1 previous similar message [ 6273.193343] Lustre: server umount lustre-OST0000 complete [ 6284.132927] Lustre: server umount lustre-MDT0000 complete [ 6288.604239] LustreError: 207471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838934 with bad export cookie 16438675871764403921 [ 6288.616422] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.622012] LustreError: 207471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6289.187129] Lustre: server umount lustre-MDT0001 complete [ 6300.027625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.496967] LustreError: 211667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.516714] LustreError: 211667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6306.274560] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6316.854057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.894043] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6330.344630] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6336.366397] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6343.170619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6343.497377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.505451] Lustre: Skipped 4 previous similar messages [ 6349.219700] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6356.894581] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6362.228548] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6362.557890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6368.221284] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6368.556394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6371.301840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.309144] LustreError: Skipped 1 previous similar message [ 6371.312800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.333844] Lustre: Skipped 2 previous similar messages [ 6377.148584] Lustre: server umount lustre-OST0000 complete [ 6385.471853] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6392.052517] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6421.006590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.627084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.653196] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6429.481562] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6438.012748] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.039442] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6440.172110] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6442.185284] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6454.622847] Lustre: server umount lustre-OST0000 complete [ 6464.829612] Lustre: server umount lustre-MDT0000 complete [ 6468.365758] LustreError: 211649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839114 with bad export cookie 16438675871764404803 [ 6468.375635] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6468.389572] LustreError: 211649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6468.649114] Lustre: server umount lustre-MDT0001 complete [ 6476.840421] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:45:21 (1788839121) [ 6483.961663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.538234] LustreError: 217641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.562967] LustreError: 217641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6488.740128] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6497.376671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.738872] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6510.082451] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6516.727187] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6523.772129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6530.518618] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6538.364277] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6542.838152] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6543.021199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6547.521962] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6547.794777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6555.049566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.355192] Lustre: lustre-OST0001: new disk, initializing [ 6555.361899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6555.370151] Lustre: 221009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6561.942150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6561.958481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6562.166776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6562.952218] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6573.276495] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6578.372531] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6578.596658] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6583.323298] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6583.508593] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6587.876503] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.884556] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.887294] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6587.887302] Lustre: Skipped 16 previous similar messages [ 6587.888310] Lustre: Skipped 6 previous similar messages [ 6587.915686] LustreError: Skipped 2 previous similar messages [ 6591.607689] Lustre: server umount lustre-OST0001 complete [ 6598.509649] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6598.847789] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6603.784195] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6604.084920] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6632.893896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.181807] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6633.193849] Lustre: Skipped 5 previous similar messages [ 6638.571623] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6638.586103] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6638.588481] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6638.617392] Lustre: Skipped 1 previous similar message [ 6639.242432] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6647.416640] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.435401] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6653.941393] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6657.797428] Lustre: server umount lustre-OST0001 complete [ 6681.501881] Lustre: server umount lustre-OST0000 complete [ 6691.946726] Lustre: server umount lustre-MDT0000 complete [ 6695.886363] LustreError: 218376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839341 with bad export cookie 16438675871764405769 [ 6695.897760] LustreError: 218376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6695.903585] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6710.239169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6710.586471] Lustre: server umount lustre-MDT0001 complete [ 6739.347599] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:49:43 (1788839383) [ 6746.130238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.352623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6746.387645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6751.359896] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6758.970175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.122577] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6759.167647] Lustre: 226258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6763.487486] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6772.020758] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6778.592297] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6788.204755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6788.422510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6796.735288] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6805.044261] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6813.476727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6813.578558] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6820.491344] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6827.208844] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6828.703936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6833.470542] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6833.683611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6839.328531] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6839.666804] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6847.524679] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9882829a9000.ost_server_uuid 50 [ 6849.403697] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9882829a9000.ost_server_uuid in IDLE state after 0 sec [ 6850.994601] Lustre: Permanently deactivating lustre-OST0001 [ 6851.051576] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6856.991158] Lustre: Permanently reactivating lustre-OST0001 [ 6857.015927] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6857.029238] Lustre: Skipped 2 previous similar messages [ 6863.842857] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.843600] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.858501] LustreError: Skipped 5 previous similar messages [ 6863.877187] Lustre: Skipped 7 previous similar messages [ 6868.230617] Lustre: server umount lustre-OST0001 complete [ 6872.040032] LustreError: 228103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.060089] LustreError: 228103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 6887.391138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6887.620726] Lustre: server umount lustre-OST0000 complete [ 6898.815886] Lustre: server umount lustre-MDT0000 complete [ 6903.059760] LustreError: 226251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839549 with bad export cookie 16438675871764407057 [ 6903.071233] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6903.089760] LustreError: 226251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6903.450199] Lustre: server umount lustre-MDT0001 complete [ 6927.598995] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:52:52 (1788839572) [ 6934.428700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.673581] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6934.697517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6934.704265] Lustre: Skipped 2 previous similar messages [ 6938.514741] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6947.443938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.541735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6947.571277] Lustre: 232956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6952.231659] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6960.069590] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6967.109360] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6976.216369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.522199] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6976.548471] Lustre: Permanently deactivating lustre-OST0000 [ 6976.566845] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6983.714603] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6984.717766] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6992.994154] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7000.617746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7000.670389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7007.510878] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7009.493186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7015.602462] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7017.383824] Lustre: Permanently reactivating lustre-OST0000 [ 7017.412337] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7017.423268] Lustre: Skipped 2 previous similar messages [ 7018.369577] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7018.384495] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000f9fa1600, cur 1788839664 deadline 1788839659 last 1788839629 [ 7018.461037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7034.856822] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7034.868854] LustreError: Skipped 3 previous similar messages [ 7038.513996] Lustre: server umount lustre-OST0001 complete [ 7048.771373] Lustre: server umount lustre-OST0000 complete [ 7059.496381] Lustre: server umount lustre-MDT0000 complete [ 7063.211465] LustreError: 232948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839709 with bad export cookie 16438675871764408737 [ 7063.212346] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7063.230955] LustreError: 232948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7063.635806] Lustre: server umount lustre-MDT0001 complete [ 7076.447143] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7079.018836] Key type lgssc unregistered [ 7079.376444] LNet: 237836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7079.385539] LNetError: 237836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7079.416221] LNet: Removed LNI 192.168.203.150@tcp [ 7080.287403] Key type .llcrypt unregistered [ 7080.293330] Key type ._llcrypt unregistered [ 7104.180615] Key type ._llcrypt registered [ 7104.183620] Key type .llcrypt registered [ 7104.305244] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7105.413668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7105.445164] alg: No test for adler32 (adler32-zlib) [ 7106.550054] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7106.869492] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7108.669703] Key type lgssc registered [ 7109.698501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7120.353459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.396319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.895317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7126.419169] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7131.114912] LustreError: 239300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.134173] LustreError: 239300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7136.007451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7136.125084] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7136.159872] Lustre: 239289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7136.169775] Lustre: Permanently deactivating lustre-MDT0001 [ 7136.189777] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7136.231815] LustreError: 239299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.464354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7140.694304] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7146.572485] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7150.594391] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7158.010939] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7165.088593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.511594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7166.505946] LustreError: 241144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7166.551903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7171.556127] LustreError: 241144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.579307] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7176.702861] LustreError: 241144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.274620] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.842112] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7187.939219] LustreError: 241195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.960497] LustreError: 241195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7188.299038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.604553] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7194.093523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7194.107064] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7195.013396] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7203.566628] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.244472] Lustre: Permanently reactivating lustre-MDT0001 [ 7208.274957] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7208.281583] Lustre: Skipped 2 previous similar messages [ 7211.350710] Lustre: lustre-MDT0001: Client 87034de4-0533-405e-ab3e-0e45174aad36 (at 192.168.203.50@tcp) reconnecting [ 7215.333711] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.360533] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7215.367837] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7215.404778] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7219.246842] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7221.354106] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7229.267505] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7229.530378] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7234.350611] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7234.564738] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7237.721090] Lustre: Permanently deactivating lustre-MDT0001 [ 7237.749511] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7237.755162] Lustre: Skipped 2 previous similar messages [ 7244.460786] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7254.500589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7254.510462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.527984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7255.216961] Lustre: server umount lustre-MDT0000 complete [ 7256.546607] LustreError: 239300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.551133] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.563290] LustreError: 239300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7256.578791] Lustre: Skipped 2 previous similar messages [ 7259.144965] LustreError: 239279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839905 with bad export cookie 7547741513163058984 [ 7259.160879] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.542586] Lustre: server umount lustre-MDT0001 complete [ 7265.899505] Lustre: server umount lustre-OST0000 complete [ 7277.203606] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:58:42 (1788839922) [ 7290.654468] Lustre: server umount lustre-OST0001 complete [ 7298.514398] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7308.194434] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7344.858389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.113578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7345.128621] Lustre: Skipped 2 previous similar messages [ 7345.151341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7345.290698] Lustre: lustre-MDT0000: new disk, initializing [ 7345.438460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7345.454498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7350.110515] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7362.797057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7362.917037] Lustre: 247558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7362.999360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7363.014242] Lustre: Skipped 1 previous similar message [ 7363.134153] Lustre: lustre-MDT0001: new disk, initializing [ 7363.265505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7363.334389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7363.353506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7368.425310] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7378.631784] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7384.517637] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7391.167374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.433428] Lustre: lustre-OST0000: new disk, initializing [ 7391.436914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7391.443540] Lustre: 249529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7391.577529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.509357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7398.538731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7398.677221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7399.014870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7409.790262] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7412.381986] LustreError: 247565:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7421.356427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7421.545695] Lustre: lustre-OST0001: new disk, initializing [ 7421.558412] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7421.575229] Lustre: 250656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7422.443135] LustreError: 247565:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7422.479253] LustreError: 250298:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7428.018962] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7428.749354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7428.763213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7428.850196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7432.559221] LustreError: 250298:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7438.136538] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7440.043895] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7442.406875] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 7444.450485] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7444.460084] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.467504] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7444.483657] Lustre: Skipped 1 previous similar message [ 7444.520300] Lustre: Skipped 1 previous similar message [ 7447.474861] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 7448.315449] Lustre: server umount lustre-OST0001 complete [ 7449.569078] LustreError: 249511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.610547] LustreError: 249511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7465.440180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.447154] LustreError: Skipped 1 previous similar message [ 7465.449442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.464527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7465.470932] Lustre: Skipped 1 previous similar message [ 7471.658317] Lustre: server umount lustre-OST0000 complete [ 7476.192654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7476.207141] LustreError: Skipped 1 previous similar message [ 7476.213288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.231197] Lustre: Skipped 1 previous similar message [ 7476.242097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.250421] Lustre: Skipped 2 previous similar messages [ 7481.941954] Lustre: server umount lustre-MDT0000 complete [ 7485.505421] LustreError: 247550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840131 with bad export cookie 7547741513163061791 [ 7485.512375] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.526839] LustreError: 247550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7492.148283] Lustre: server umount lustre-MDT0001 complete [ 7503.758391] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7506.573650] Key type lgssc unregistered [ 7506.905417] LNet: 252688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7506.920190] LNetError: 252688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7506.945723] LNet: Removed LNI 192.168.203.150@tcp [ 7508.069891] Key type .llcrypt unregistered [ 7508.072342] Key type ._llcrypt unregistered [ 7550.595764] Key type ._llcrypt registered [ 7550.599173] Key type .llcrypt registered [ 7550.691677] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7551.992980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7552.068639] alg: No test for adler32 (adler32-zlib) [ 7553.203852] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7553.465921] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7555.119978] Key type lgssc registered [ 7556.100387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7564.594373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7564.617335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.064153] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7566.108676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7566.342377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.497882] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7584.013707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.173842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7584.236207] Lustre: 254896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7584.488238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.062663] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7598.749688] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7605.455178] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7612.972175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7613.256738] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7613.492297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7620.423406] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7620.787762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7628.819573] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7687.147074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7687.160052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.175463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.663019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.676488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7690.274589] Lustre: server umount lustre-OST0000 complete [ 7694.222811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7706.364113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7706.643699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.718373] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7707.747371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7707.750628] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7713.001308] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7720.123325] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7758.829941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.845702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7758.855913] Lustre: Skipped 1 previous similar message [ 7763.939591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7763.944578] Lustre: Skipped 1 previous similar message [ 7764.637407] Lustre: server umount lustre-OST0000 complete [ 7769.058409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.075611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.083401] Lustre: Skipped 1 previous similar message [ 7779.302583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7779.315959] Lustre: Skipped 3 previous similar messages [ 7782.368914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7782.607867] Lustre: server umount lustre-MDT0000 complete [ 7786.513197] LustreError: 254888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840432 with bad export cookie 8068259805964704467 [ 7786.518935] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7786.533983] LustreError: 254888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7786.910944] Lustre: server umount lustre-MDT0001 complete [ 7797.483482] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7800.253880] Key type lgssc unregistered [ 7800.528803] LNet: 260325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7800.540140] LNetError: 260325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7800.568760] LNet: Removed LNI 192.168.203.150@tcp [ 7801.504393] Key type .llcrypt unregistered [ 7801.507315] Key type ._llcrypt unregistered [ 7826.370253] Key type ._llcrypt registered [ 7826.377139] Key type .llcrypt registered [ 7826.505533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7827.633918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7827.693813] alg: No test for adler32 (adler32-zlib) [ 7828.777699] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7829.084981] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7830.847327] Key type lgssc registered [ 7832.045400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7840.048618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7840.088578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.566698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.996568] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7850.983840] LustreError: 261751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.013137] LustreError: 261751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7854.785744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.039586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7858.785725] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7866.498926] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7872.018118] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7878.585632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7878.928649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7884.505880] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7886.135792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7891.007476] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7916.000710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7916.007565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.027894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7917.031140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7917.034141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.034624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7918.904499] Lustre: server umount lustre-OST0000 complete [ 7926.755101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7926.767736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.779797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7926.785387] Lustre: Skipped 1 previous similar message [ 7928.659468] Lustre: server umount lustre-MDT0000 complete [ 7932.387982] LustreError: 262479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.412962] LustreError: 262479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7932.480317] LustreError: 262487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840578 with bad export cookie 2455226452991755676 [ 7932.486423] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7932.493814] LustreError: 262487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7932.848801] Lustre: server umount lustre-MDT0001 complete [ 7945.364533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7948.026699] Key type lgssc unregistered [ 7948.223383] LNet: 266055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7948.229794] LNetError: 266055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7948.246675] LNet: Removed LNI 192.168.203.150@tcp [ 7948.967684] Key type .llcrypt unregistered [ 7948.969786] Key type ._llcrypt unregistered [ 7970.271242] Key type ._llcrypt registered [ 7970.275275] Key type .llcrypt registered [ 7970.404803] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7971.251043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7971.346439] alg: No test for adler32 (adler32-zlib) [ 7972.426794] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 7972.649770] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7974.320135] Key type lgssc registered [ 7975.469116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7982.165595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7982.190131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7983.602753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7987.563806] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7993.838349] LustreError: 267273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.858501] LustreError: 267273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7995.064219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.419632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.978393] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8007.372988] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8012.539554] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8018.685876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8019.145960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8024.169109] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8028.345970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8031.422519] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.000952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.008829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.016664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.052702] Lustre: Skipped 1 previous similar message [ 8047.009993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8047.017521] Lustre: Skipped 1 previous similar message [ 8049.665189] Lustre: server umount lustre-OST0000 complete [ 8056.801807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8056.811201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.837135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.847616] Lustre: Skipped 1 previous similar message [ 8058.904651] Lustre: server umount lustre-MDT0000 complete [ 8062.442070] LustreError: 267270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.471318] LustreError: 267270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8062.697878] LustreError: 267255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840708 with bad export cookie 15956959545426076990 [ 8062.706867] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.708106] LustreError: 267255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8063.125826] Lustre: server umount lustre-MDT0001 complete [ 8074.005826] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8077.090763] Key type lgssc unregistered [ 8077.413450] LNet: 270949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8077.431858] LNetError: 270949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8077.449897] LNet: Removed LNI 192.168.203.150@tcp [ 8078.449701] Key type .llcrypt unregistered [ 8078.452859] Key type ._llcrypt unregistered [ 8103.650158] Key type ._llcrypt registered [ 8103.653449] Key type .llcrypt registered [ 8103.754628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8104.729767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8104.863112] alg: No test for adler32 (adler32-zlib) [ 8106.026587] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8106.262338] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8107.967276] Key type lgssc registered [ 8109.102317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8116.666676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8116.681886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.113762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.823596] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8128.489369] LustreError: 272361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8128.522871] LustreError: 272361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8131.241317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.582541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8135.666204] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8142.353653] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.771988] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8154.120385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.576616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.166866] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8163.838785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8169.858289] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8194.543440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8194.552789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.569437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8199.651137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8199.660955] Lustre: Skipped 1 previous similar message [ 8200.762924] Lustre: server umount lustre-OST0000 complete [ 8208.352487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.365316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.383478] Lustre: Skipped 1 previous similar message [ 8208.388127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8208.396421] Lustre: Skipped 1 previous similar message [ 8209.889866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.305957] Lustre: server umount lustre-MDT0000 complete [ 8214.808205] LustreError: 274204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840860 with bad export cookie 8053214274532110389 [ 8214.812507] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8214.830091] LustreError: 274204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8215.151979] Lustre: server umount lustre-MDT0001 complete [ 8225.630785] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8228.380528] Key type lgssc unregistered [ 8228.783953] LNet: 276666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8228.797778] LNetError: 276666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8228.816651] LNet: Removed LNI 192.168.203.150@tcp [ 8229.861579] Key type .llcrypt unregistered [ 8229.865654] Key type ._llcrypt unregistered [ 8251.417560] Key type ._llcrypt registered [ 8251.419759] Key type .llcrypt registered [ 8251.513533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8252.830822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8252.859324] alg: No test for adler32 (adler32-zlib) [ 8253.962429] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8254.375150] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8256.071160] Key type lgssc registered [ 8257.424639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8266.181758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8266.242204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.830582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8272.605393] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8276.961419] LustreError: 277897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.991568] LustreError: 277897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8281.346504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.785548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8286.535228] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8294.434948] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8300.809754] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8307.672598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.070229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.181452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8314.334870] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8322.330872] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8337.888420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8337.901963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.923190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8341.986929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8341.988629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.010781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.914979] Lustre: server umount lustre-OST0000 complete [ 8348.647191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8348.659425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.689406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8348.698480] Lustre: Skipped 1 previous similar message [ 8352.227259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8352.250493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.448765] Lustre: server umount lustre-MDT0000 complete [ 8357.347137] LustreError: 277902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.367809] LustreError: 277902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8359.227863] LustreError: 277864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841005 with bad export cookie 6503781888197278847 [ 8359.237508] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.256326] LustreError: 277864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8359.803413] Lustre: server umount lustre-MDT0001 complete [ 8373.106351] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8376.356650] Key type lgssc unregistered [ 8376.617664] LNet: 281600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8376.630700] LNetError: 281600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8376.660695] LNet: Removed LNI 192.168.203.150@tcp [ 8377.642525] Key type .llcrypt unregistered [ 8377.646743] Key type ._llcrypt unregistered [ 8391.286302] Key type ._llcrypt registered [ 8391.292822] Key type .llcrypt registered [ 8391.539816] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8409.794600] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8426.957726] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8427.861065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8427.897224] alg: No test for adler32 (adler32-zlib) [ 8429.096637] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8429.350492] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8431.087210] Key type lgssc registered [ 8432.436539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8470.857703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8470.896727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.305640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8472.382266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8472.514631] Lustre: lustre-MDT0000: new disk, initializing [ 8472.619123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.642350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8477.552363] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8488.562382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.659630] Lustre: 285154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8488.710262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8488.715143] Lustre: Skipped 1 previous similar message [ 8488.779696] Lustre: lustre-MDT0001: new disk, initializing [ 8488.865966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8488.903113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8488.922926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8493.335062] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8503.332877] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8509.767283] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.551156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8516.994591] Lustre: lustre-OST0000: new disk, initializing [ 8517.005035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8517.013333] Lustre: 287122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8517.120429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.489189] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8524.386924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8524.412686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8524.580336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8532.836398] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.697363] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8537.900068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8542.669520] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8542.950507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8545.253777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8545.253939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8545.272129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8545.278725] Lustre: Skipped 1 previous similar message [ 8550.384906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8550.400692] Lustre: Skipped 1 previous similar message [ 8551.593285] Lustre: server umount lustre-OST0000 complete [ 8560.611941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8560.618957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8560.648712] Lustre: Skipped 2 previous similar messages [ 8560.663172] Lustre: Skipped 1 previous similar message [ 8562.977638] Lustre: server umount lustre-MDT0000 complete [ 8567.437065] LustreError: 285145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841213 with bad export cookie 10450593131092208297 [ 8567.440714] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.455971] LustreError: 285145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8567.968798] Lustre: server umount lustre-MDT0001 complete [ 8577.537057] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:20:22 (1788841222) [ 8584.888594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.511820] LustreError: 289534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.535610] LustreError: 289534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8585.633377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.823205] LustreError: 289535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.336259] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8595.949489] LustreError: 289534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.197144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.632628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.601708] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8613.840407] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8620.535484] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8627.255934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8627.487000] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.673382] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8642.429253] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8657.376569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8657.393628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8657.410081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8661.472815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8661.475090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8661.485313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.491365] Lustre: Skipped 1 previous similar message [ 8663.622246] Lustre: server umount lustre-OST0000 complete [ 8671.724484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.732769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.758019] Lustre: Skipped 1 previous similar message [ 8671.785573] Lustre: Skipped 1 previous similar message [ 8674.880271] Lustre: server umount lustre-MDT0000 complete [ 8676.851136] LustreError: 289534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.801511] LustreError: 289514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841325 with bad export cookie 10450593131092209179 [ 8679.813231] LustreError: 289514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8679.814632] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.232390] Lustre: server umount lustre-MDT0001 complete [ 8691.082329] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 8693.578971] Key type lgssc unregistered [ 8693.844922] LNet: 293091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8693.867151] LNetError: 293091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8693.899395] LNet: Removed LNI 192.168.203.150@tcp [ 8695.027166] Key type .llcrypt unregistered [ 8695.031937] Key type ._llcrypt unregistered [ 8726.493357] Key type ._llcrypt registered [ 8726.500562] Key type .llcrypt registered [ 8726.688587] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8743.160661] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 8745.468099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8745.544532] alg: No test for adler32 (adler32-zlib) [ 8746.663686] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 8746.968301] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 8748.630359] Key type lgssc registered [ 8750.425404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8802.850934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8802.898476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.230206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8804.267249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8804.364344] Lustre: lustre-MDT0000: new disk, initializing [ 8804.474827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.494104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8809.034634] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8820.454600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.620521] Lustre: 296917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8820.721453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8820.733939] Lustre: Skipped 1 previous similar message [ 8820.890568] Lustre: lustre-MDT0001: new disk, initializing [ 8821.055835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8821.106082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8821.135320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8826.323893] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8836.421227] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8843.126243] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8850.619143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8850.951247] Lustre: lustre-OST2710: new disk, initializing [ 8850.961252] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8850.974491] Lustre: 298887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8851.065189] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8857.154927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8857.165165] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8857.309530] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8857.349777] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8868.082449] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8876.457655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8876.657393] Lustre: lustre-OST03e8: new disk, initializing [ 8876.666036] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8876.676985] Lustre: 299952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8876.852951] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8883.557801] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 8883.866146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8883.882272] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8883.992949] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8891.914508] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8899.837720] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8900.073466] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8904.728996] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8905.029434] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8909.976136] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8910.184843] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8915.347287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8915.657409] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8923.150256] Lustre: server umount lustre-MDT0000 complete [ 8923.618081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8923.631122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8925.162427] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8925.178836] Lustre: Skipped 2 previous similar messages [ 8927.102140] LustreError: 296911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841573 with bad export cookie 9406829426697132387 [ 8927.108109] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8927.119295] LustreError: 296911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8927.574882] Lustre: server umount lustre-MDT0001 complete [ 8941.199231] Lustre: server umount lustre-OST2710 complete [ 8954.275969] Lustre: server umount lustre-OST03e8 complete [ 8976.747833] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 8990.300196] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9033.098462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9033.368988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9033.415778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9033.578296] Lustre: lustre-MDT0000: new disk, initializing [ 9033.655887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9033.681249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9038.388969] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9049.636291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.771975] Lustre: 305202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9049.831732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9049.834833] Lustre: Skipped 1 previous similar message [ 9049.914470] Lustre: lustre-MDT0001: new disk, initializing [ 9050.024296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9050.060721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9050.074467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9055.096657] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9064.465869] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9069.690698] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9077.191683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9077.497318] Lustre: lustre-OST0000: new disk, initializing [ 9077.505171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9077.519361] Lustre: 307176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9077.656799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.286245] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9087.059630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9087.077220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9087.164905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9094.302822] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9099.361473] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9099.616995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9104.065947] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9104.272531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9107.939329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.950546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.954553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9107.960011] Lustre: Skipped 1 previous similar message [ 9113.072390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9113.076320] Lustre: Skipped 2 previous similar messages [ 9118.178919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9120.223922] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9120.438444] Lustre: server umount lustre-OST0000 complete [ 9126.881234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9126.892324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.939629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9126.949596] Lustre: Skipped 1 previous similar message [ 9130.451490] Lustre: server umount lustre-MDT0000 complete [ 9133.539483] LustreError: 305214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.556881] LustreError: 305214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9134.812525] LustreError: 307174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841780 with bad export cookie 9406829426697134060 [ 9134.818582] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9134.829799] LustreError: 307174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9135.129276] Lustre: server umount lustre-MDT0001 complete [ 9145.928707] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:29:50 (1788841790) [ 9147.696892] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9149.644349] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:29:54 (1788841794) [ 9154.203626] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9187.284711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.570571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9187.594251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9187.796828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9193.394819] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9203.711741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9203.876423] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9203.930632] Lustre: 311037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9210.167754] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9233.776653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9234.008694] Lustre: Denying initial registration attempt from nid 192.168.203.150@tcp, specified as failover [ 9234.019596] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9234.032198] LustreError: 312491:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9234.044957] LustreError: 312491:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9234.055192] LustreError: 312491:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9234.143375] Lustre: server umount lustre-OST0000 complete [ 9234.150910] LustreError: 312491:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9242.606216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.620584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.628218] Lustre: Skipped 2 previous similar messages [ 9242.650074] Lustre: Skipped 3 previous similar messages [ 9247.685364] Lustre: server umount lustre-MDT0000 complete [ 9247.730517] LustreError: 311048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9251.700108] LustreError: 311028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841897 with bad export cookie 9406829426697134949 [ 9251.702534] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9251.723289] LustreError: 311028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9252.162362] Lustre: server umount lustre-MDT0001 complete [ 9264.124201] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9267.150671] Key type lgssc unregistered [ 9267.520680] LNet: 313610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9267.532669] LNetError: 313610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9267.575301] LNet: Removed LNI 192.168.203.150@tcp [ 9268.565548] Key type .llcrypt unregistered [ 9268.568683] Key type ._llcrypt unregistered [ 9279.847302] Key type ._llcrypt registered [ 9279.850186] Key type .llcrypt registered [ 9279.982560] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9281.143614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9281.181831] alg: No test for adler32 (adler32-zlib) [ 9282.266436] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 9282.491574] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9284.159297] Key type lgssc registered [ 9285.635295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9332.735387] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9341.724771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9341.746588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.025911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9343.060292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9343.224875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.060301] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9357.489569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.676168] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9357.757898] Lustre: 316371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9358.131480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9362.657377] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9370.638973] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9377.843485] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9400.694974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9400.935225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9401.168755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.581315] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9415.921555] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9419.238297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9419.251821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.261226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9419.265448] Lustre: Skipped 1 previous similar message [ 9419.289793] Lustre: Skipped 1 previous similar message [ 9424.212847] Lustre: server umount lustre-OST0000 complete [ 9429.473017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.493477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.503087] Lustre: Skipped 1 previous similar message [ 9434.594184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9434.601886] Lustre: Skipped 1 previous similar message [ 9434.983736] Lustre: server umount lustre-MDT0000 complete [ 9438.963144] LustreError: 316362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842085 with bad export cookie 9802972511536202433 [ 9438.981485] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9439.508120] Lustre: server umount lustre-MDT0001 complete [ 9450.160739] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9452.682914] Key type lgssc unregistered [ 9452.969676] LNet: 319947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9452.982167] LNetError: 319947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9453.008542] LNet: Removed LNI 192.168.203.150@tcp [ 9453.772274] Key type .llcrypt unregistered [ 9453.774404] Key type ._llcrypt unregistered [ 9477.883797] Key type ._llcrypt registered [ 9477.886567] Key type .llcrypt registered [ 9477.962727] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9478.878969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9478.926402] alg: No test for adler32 (adler32-zlib) [ 9479.945707] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 9480.213364] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9481.887405] Key type lgssc registered [ 9482.988713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9490.821651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9490.867867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9492.227859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.371801] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9502.690379] LustreError: 321358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.713848] LustreError: 321358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9504.612507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.960804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9509.713882] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9517.997900] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9524.248631] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9532.852035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9533.421039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9540.582307] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9549.000583] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9560.043691] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9560.046902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9560.068672] Lustre: Skipped 1 previous similar message [ 9560.087200] Lustre: Skipped 2 previous similar messages [ 9562.065902] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 9565.160304] LustreError: 323381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.172342] LustreError: 323381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9567.011269] Lustre: server umount lustre-MDT0000 complete [ 9567.169771] LustreError: 321352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9570.272236] LustreError: 321359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9570.282991] LustreError: 321359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9570.963798] LustreError: 321339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842217 with bad export cookie 1151975296893425080 [ 9570.984994] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9570.998215] LustreError: 321339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9572.294640] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [ 9575.392160] LustreError: 323381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9575.393064] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9575.417651] LustreError: 323381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9575.442315] Lustre: Skipped 1 previous similar message [ 9575.448113] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9577.344900] Lustre: server umount lustre-MDT0001 complete [ 9584.705454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9594.416591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9596.639168] LustreError: 324940:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9596.652824] LustreError: 324940:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000009d8f6a82 x1875736921145984/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788842242 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9596.680774] LustreError: 324940:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9596.832048] LustreError: 320668:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000424d0241 x1875736921146240/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9597.136703] LustreError: 323192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9597.256223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.319172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9602.789286] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9605.417914] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9613.437501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.758317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.823937] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9618.610939] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9628.854597] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9630.717240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9638.498549] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9640.729914] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9646.577715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9646.583433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9646.590089] Lustre: Skipped 1 previous similar message [ 9652.649320] Lustre: server umount lustre-OST0000 complete [ 9659.873457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9659.887644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.905695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.911299] Lustre: Skipped 3 previous similar messages [ 9663.995780] Lustre: server umount lustre-MDT0000 complete [ 9667.043450] LustreError: 324957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.059609] LustreError: 324957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9668.042502] LustreError: 321339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842314 with bad export cookie 1151975296893432227 [ 9668.045659] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9668.073515] LustreError: 321339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9668.437929] Lustre: server umount lustre-MDT0001 complete [ 9680.751445] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9684.139220] Key type lgssc unregistered [ 9684.518386] LNet: 327626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9684.529624] LNetError: 327626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9684.544881] LNet: Removed LNI 192.168.203.150@tcp [ 9685.616337] Key type .llcrypt unregistered [ 9685.621494] Key type ._llcrypt unregistered [ 9712.790839] Key type ._llcrypt registered [ 9712.795320] Key type .llcrypt registered [ 9712.955678] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 9714.196610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9714.239549] alg: No test for adler32 (adler32-zlib) [ 9715.453723] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [ 9715.687148] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 9717.375184] Key type lgssc registered [ 9718.469835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9727.477297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9727.528915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9729.045779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9733.964842] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9739.237396] LustreError: 329041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.286896] LustreError: 329041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9744.002228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.464227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9749.295400] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9758.344305] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9765.355896] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.957061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9769.961255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9769.973818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9770.020951] Lustre: Skipped 1 previous similar message [ 9771.046292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9771.064169] Lustre: Skipped 2 previous similar messages [ 9775.654136] Lustre: server umount lustre-MDT0000 complete [ 9776.096611] LustreError: 329035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.116741] LustreError: 329035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9780.066815] LustreError: 329021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842426 with bad export cookie 11464528127605398676 [ 9780.082224] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9780.099342] LustreError: 329021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9780.482628] Lustre: server umount lustre-MDT0001 complete [ 9796.556813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9796.779348] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9796.813429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9797.102973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9801.749302] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9811.006580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9811.185731] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9811.244949] Lustre: 331629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9811.548650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9816.129119] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9823.364665] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9828.709306] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9838.317068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9838.485151] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9838.494125] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9838.501828] LustreError: 333511:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9838.512044] LustreError: 333511:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9838.515822] LustreError: 333511:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9838.562945] Lustre: server umount lustre-OST0000 complete [ 9838.567821] LustreError: 333511:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9848.882234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9849.221620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9849.373050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.597588] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9857.668498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9863.644962] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9871.754972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9871.848061] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9871.906840] Lustre: lustre-OST0001: new disk, initializing [ 9871.910805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9871.916386] Lustre: 335054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9872.002591] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9877.629591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9877.645823] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9877.793624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9878.746551] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 9885.914480] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9893.346617] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9893.367494] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.397541] Lustre: Skipped 1 previous similar message [ 9893.400316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9894.811582] Lustre: server umount lustre-OST0001 complete [ 9898.469405] LustreError: 335062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.514875] LustreError: 335062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9903.590180] LustreError: 333976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.600809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.604727] LustreError: 333976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9903.608270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9903.628367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9903.630015] LustreError: Skipped 1 previous similar message [ 9903.642577] Lustre: Skipped 1 previous similar message [ 9905.500612] Lustre: server umount lustre-OST0000 complete [ 9913.824500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9913.830418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9913.839834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9913.839844] Lustre: Skipped 1 previous similar message [ 9913.873459] Lustre: Skipped 2 previous similar messages [ 9915.640544] Lustre: server umount lustre-MDT0000 complete [ 9918.944496] LustreError: 332789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9918.977434] LustreError: 332789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9919.622549] LustreError: 331620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842565 with bad export cookie 11464528127605399229 [ 9919.634548] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9919.640252] LustreError: 331620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9919.956745] Lustre: server umount lustre-MDT0001 complete [ 9929.814247] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 9932.744592] Key type lgssc unregistered [ 9932.988719] LNet: 337010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9933.002313] LNetError: 337010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9933.026878] LNet: Removed LNI 192.168.203.150@tcp [ 9934.187522] Key type .llcrypt unregistered [ 9934.191476] Key type ._llcrypt unregistered [10011.433875] Key type ._llcrypt registered [10011.436537] Key type .llcrypt registered [10011.565211] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [10026.429973] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10027.688786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10027.735144] alg: No test for adler32 (adler32-zlib) [10028.902338] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [10029.171410] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10030.879168] Key type lgssc registered [10031.988781] Lustre: Echo OBD driver; http://www.lustre.org/ [10069.881824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10069.923902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.220190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10071.269848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10071.424181] Lustre: lustre-MDT0000: new disk, initializing [10071.603877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10071.625812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10076.129775] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10087.395191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.609392] Lustre: 341943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10087.665812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10087.670498] Lustre: Skipped 1 previous similar message [10087.734759] Lustre: lustre-MDT0001: new disk, initializing [10087.855615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10087.919066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10087.935539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10092.498386] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10103.164623] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10109.481588] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10117.501842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.877160] Lustre: lustre-OST0000: new disk, initializing [10117.883719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10117.901602] Lustre: 343910:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10118.087636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.291282] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10126.969818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10126.986948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10127.170059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10135.020877] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10139.940256] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10140.217920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10144.583571] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10144.836033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10147.811461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.822037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10147.823778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10152.947356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10152.957901] Lustre: Skipped 2 previous similar messages [10158.060946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.063838] Lustre: Skipped 1 previous similar message [10161.119207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10161.271841] Lustre: server umount lustre-OST0000 complete [10168.289051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.329478] Lustre: Skipped 1 previous similar message [10168.350838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10171.347110] Lustre: server umount lustre-MDT0000 complete [10175.186894] LustreError: 342746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842821 with bad export cookie 12244336633327377987 [10175.192609] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10175.202255] LustreError: 342746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10175.620477] Lustre: server umount lustre-MDT0001 complete [10184.857519] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:47:10 (1788842830) [10193.227162] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:47:18 (1788842838) [10201.999249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10202.506939] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10202.534093] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10202.611413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10206.866572] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10207.715412] LustreError: 346668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.840904] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10215.467164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.862813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10219.861419] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10227.287440] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10234.902308] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10244.831215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10245.237076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10252.197771] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10259.773316] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10267.592451] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10269.552429] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10271.506321] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10273.787915] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10275.935313] Lustre: Failing over lustre-MDT0000 [10276.283271] Lustre: server umount lustre-MDT0000 complete [10276.322171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10276.335485] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.349591] Lustre: Skipped 3 previous similar messages [10276.381404] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10281.445631] LustreError: 347397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.478704] LustreError: 347397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10291.683701] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10291.699281] LustreError: 346667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10292.577359] Lustre: 340165:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842922/real 1788842922] req@00000000b190c196 x1875737496854144/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788842938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10292.602674] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10295.977950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.307216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10303.412764] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10303.922243] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10308.286923] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10308.607750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10308.642298] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10308.684932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10318.338701] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10320.381588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10324.078203] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10326.183333] Lustre: Failing over lustre-MDT0000 [10326.473364] Lustre: server umount lustre-MDT0000 complete [10329.058332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.070294] Lustre: Skipped 1 previous similar message [10329.070664] LustreError: 346662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10329.074182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10329.105019] LustreError: 346662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10345.447971] Lustre: 340164:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842975/real 1788842975] req@00000000ebaae88b x1875737496876032/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788842991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10345.485307] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10345.608066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.892659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10355.928854] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10356.155728] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10361.346277] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10361.377958] Lustre: Skipped 2 previous similar messages [10361.454489] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10361.506830] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10361.568715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10373.249789] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10375.601908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10386.914666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.917361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10386.926579] Lustre: Skipped 2 previous similar messages [10386.930072] Lustre: Skipped 2 previous similar messages [10389.025175] Lustre: server umount lustre-MDT0000 complete [10392.032376] LustreError: 347397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.059481] LustreError: 347397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10393.227596] LustreError: 346647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843039 with bad export cookie 12244336633327380731 [10393.241732] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.253216] LustreError: 346647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10393.795151] Lustre: server umount lustre-MDT0001 complete [10409.027064] Lustre: server umount lustre-OST0000 complete [10425.248674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.743697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10430.234151] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10438.326697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.929399] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10450.138332] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10456.107599] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10462.561517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10469.109172] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10470.138435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10476.805705] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10484.336311] Lustre: server umount lustre-OST0000 complete [10489.834204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10489.837695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.853268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.859235] Lustre: Skipped 1 previous similar message [10499.273475] Lustre: server umount lustre-MDT0000 complete [10500.065908] Lustre: 340166:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843130/real 1788843130] req@0000000093f4c5a3 x1875737496929920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788843146 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10500.081574] LustreError: 353014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10500.090853] LustreError: 353014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10504.548301] LustreError: 352999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843150 with bad export cookie 12244336633327381438 [10504.550907] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10504.559956] LustreError: 352999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10505.220707] Lustre: server umount lustre-MDT0001 complete [10518.554338] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10521.226537] Key type lgssc unregistered [10521.501928] LNet: 356551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10521.508637] LNetError: 356551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10521.528300] LNet: Removed LNI 192.168.203.150@tcp [10522.563601] Key type .llcrypt unregistered [10522.571491] Key type ._llcrypt unregistered [10550.379950] Key type ._llcrypt registered [10550.382620] Key type .llcrypt registered [10550.473650] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10551.470956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10551.497538] alg: No test for adler32 (adler32-zlib) [10552.631867] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [10552.941573] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10554.663363] Key type lgssc registered [10555.962577] Lustre: Echo OBD driver; http://www.lustre.org/ [10563.878832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10563.925448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10565.342709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10569.112695] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10575.847168] LustreError: 358063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10575.874612] LustreError: 358063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10577.799867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10578.192125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10583.293728] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10591.999715] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10599.405931] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10606.741638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10607.390986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10610.482816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10613.840592] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10621.920192] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10638.506972] Lustre: server umount lustre-MDT0000 complete [10639.330798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10639.333158] LustreError: 358060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.349131] Lustre: Skipped 1 previous similar message [10639.377766] LustreError: 358060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10642.246912] LustreError: 358044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843288 with bad export cookie 18341394576412458380 [10642.256301] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.257666] LustreError: 358044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10642.758413] Lustre: server umount lustre-MDT0001 complete [10657.113674] Lustre: server umount lustre-OST0000 complete [10672.818623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.284565] LustreError: 361798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10673.381982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10678.152037] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10678.752419] LustreError: 361799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.880655] LustreError: 361798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10687.358129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10687.932058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10692.727073] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10701.185638] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10707.295318] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10715.072992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.539316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10718.661513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10722.755348] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10729.999169] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10742.242953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.252139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10742.258019] Lustre: Skipped 2 previous similar messages [10742.275508] Lustre: Skipped 2 previous similar messages [10747.363015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10747.375789] Lustre: Skipped 2 previous similar messages [10748.495858] Lustre: server umount lustre-MDT0000 complete [10752.063103] LustreError: 361778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843398 with bad export cookie 18341394576412459493 [10752.074223] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10752.080327] LustreError: 361778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10752.462219] Lustre: server umount lustre-MDT0001 complete [10766.755265] Lustre: server umount lustre-OST0000 complete [10784.069228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10784.748550] LustreError: 365711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.864688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10790.217920] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10800.631707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10806.049826] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10814.261349] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10820.659262] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10828.815040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10829.288764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10829.295666] Lustre: Skipped 1 previous similar message [10830.325530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10835.552222] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [10843.937942] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10850.788431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10850.794941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10850.806166] Lustre: Skipped 1 previous similar message [10850.811713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10851.817172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10855.238568] Lustre: server umount lustre-OST0000 complete [10862.058721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.066401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10862.080521] Lustre: Skipped 2 previous similar messages [10862.083777] Lustre: Skipped 2 previous similar messages [10865.826842] Lustre: server umount lustre-MDT0000 complete [10867.175433] LustreError: 366814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.222144] LustreError: 366814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10869.872869] LustreError: 365692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843515 with bad export cookie 18341394576412460515 [10869.878722] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10869.886145] LustreError: 365692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10870.265132] Lustre: server umount lustre-MDT0001 complete [10881.759478] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10884.950621] Key type lgssc unregistered [10885.215870] LNet: 369243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10885.229752] LNetError: 369243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10885.247776] LNet: Removed LNI 192.168.203.150@tcp [10886.033536] Key type .llcrypt unregistered [10886.040361] Key type ._llcrypt unregistered [10911.968572] Key type ._llcrypt registered [10911.970869] Key type .llcrypt registered [10912.098572] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [10913.697343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10913.749746] alg: No test for adler32 (adler32-zlib) [10914.792755] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [10915.020542] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [10916.695166] Key type lgssc registered [10917.773791] Lustre: Echo OBD driver; http://www.lustre.org/ [10925.735923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10925.773789] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.783843] LustreError: 370679:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10925.800334] LustreError: 370679:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10925.805069] LustreError: 370679:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10925.810308] LustreError: 370679:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10925.823878] LustreError: 370679:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10934.860966] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10934.866378] LustreError: 371077:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10934.891834] LustreError: 371077:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10934.905527] LustreError: 371077:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10934.914988] LustreError: 371077:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10934.931625] LustreError: 371077:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10957.276906] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [10960.141662] Key type lgssc unregistered [10960.452602] LNet: 372079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10960.459346] LNetError: 372079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10960.491672] LNet: Removed LNI 192.168.203.150@tcp [10961.402523] Key type .llcrypt unregistered [10961.405246] Key type ._llcrypt unregistered [10982.217695] Key type ._llcrypt registered [10982.223256] Key type .llcrypt registered [10982.335919] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [11000.908024] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11001.552903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11001.561761] alg: No test for adler32 (adler32-zlib) [11002.857295] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [11003.119876] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11004.775190] Key type lgssc registered [11005.848624] Lustre: Echo OBD driver; http://www.lustre.org/ [11042.381435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11042.410402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.727272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11043.774055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11043.879866] Lustre: lustre-MDT0000: new disk, initializing [11044.038853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11044.065634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11048.790935] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11059.494986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.597597] Lustre: 375249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11059.677504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11059.685658] Lustre: Skipped 1 previous similar message [11059.801360] Lustre: lustre-MDT0001: new disk, initializing [11059.911584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11059.975497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11059.988491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11064.647429] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11074.523574] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11080.397425] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11086.746314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11086.979763] Lustre: lustre-OST0000: new disk, initializing [11086.983797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11086.993893] Lustre: 377215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11087.083281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11092.999211] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11094.608020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11094.623877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11094.741300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11102.348514] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11106.872679] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11107.137083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11112.014333] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11112.263114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11115.497255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11115.504443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.511035] LustreError: Skipped 1 previous similar message [11115.512292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11115.547490] Lustre: Skipped 1 previous similar message [11120.660433] Lustre: server umount lustre-OST0000 complete [11125.731133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.749564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11125.762465] Lustre: Skipped 2 previous similar messages [11130.905833] Lustre: server umount lustre-MDT0000 complete [11134.831643] LustreError: 375242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843780 with bad export cookie 12494674214277255193 [11134.842848] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11134.854905] LustreError: 375242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11135.202296] Lustre: server umount lustre-MDT0001 complete [11144.617292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11146.195259] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:03:11 (1788843791) [11156.239845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.795950] LustreError: 379677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11156.830250] LustreError: 379677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11156.933478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11162.080640] LustreError: 379678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.090562] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11166.176849] LustreError: 379677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.300574] LustreError: 379678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11173.779272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.406264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11179.264805] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11187.260959] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11194.175804] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11201.872118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.396942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11208.369212] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11216.264820] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11223.687565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11223.845615] Lustre: lustre-OST0001: new disk, initializing [11223.855850] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11223.870575] Lustre: 382512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11223.984470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11229.988370] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11231.329293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11231.337453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11231.445312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11240.134633] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11247.074893] Lustre: lustre-OST0001: Not available for connect from 192.168.203.50@tcp (stopping) [11247.077504] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11247.082585] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11247.082605] Lustre: Skipped 1 previous similar message [11247.128395] LustreError: Skipped 1 previous similar message [11250.399128] Lustre: server umount lustre-OST0001 complete [11252.142570] LustreError: 381500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.437378] LustreError: 381500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.455044] LustreError: 381500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11271.141373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11271.149246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.158328] Lustre: Skipped 1 previous similar message [11271.162832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.167455] Lustre: Skipped 2 previous similar messages [11274.596223] Lustre: server umount lustre-OST0000 complete [11281.895432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11281.901543] LustreError: Skipped 1 previous similar message [11281.907604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.915302] Lustre: Skipped 1 previous similar message [11281.918306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11281.922402] Lustre: Skipped 2 previous similar messages [11286.156869] Lustre: server umount lustre-MDT0000 complete [11288.033370] LustreError: 379678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.071871] LustreError: 379678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11290.807061] LustreError: 382511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843936 with bad export cookie 12494674214277256075 [11290.812783] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11290.816281] LustreError: 382511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11297.386984] Lustre: server umount lustre-MDT0001 complete [11310.051770] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11313.104485] Key type lgssc unregistered [11313.478618] LNet: 384500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11313.485711] LNetError: 384500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11313.500954] LNet: Removed LNI 192.168.203.150@tcp [11314.399848] Key type .llcrypt unregistered [11314.402252] Key type ._llcrypt unregistered [11369.027761] Key type ._llcrypt registered [11369.030365] Key type .llcrypt registered [11369.203371] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11370.550237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11370.614284] alg: No test for adler32 (adler32-zlib) [11371.679977] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [11371.903763] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11373.639305] Key type lgssc registered [11374.854226] Lustre: Echo OBD driver; http://www.lustre.org/ [11383.852791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11383.881452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.243259] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11385.269634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11385.514103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.103731] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11401.335103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11401.503902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11401.562272] Lustre: 387297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11401.725404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11401.997686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11407.345991] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11415.328511] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11421.515919] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11427.820240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11427.820540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11427.829404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11427.846069] Lustre: Skipped 1 previous similar message [11429.719774] Lustre: server umount lustre-MDT0000 complete [11433.899782] LustreError: 387288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844079 with bad export cookie 1197417526151144641 [11433.911184] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11433.912783] LustreError: 387288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11434.192628] Lustre: server umount lustre-MDT0001 complete [11444.655445] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:08:09 (1788844089) [11451.270965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.825367] LustreError: 389810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.844802] LustreError: 389810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11451.888584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11456.164501] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11456.995489] LustreError: 389811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11462.113381] LustreError: 389810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.504792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.932419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11469.452743] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11476.711265] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11482.176269] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11489.578519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11489.884909] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11490.197126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11497.543925] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11507.490704] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11516.099859] Lustre: Permanently deactivating lustre-OST0000 [11516.122980] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11525.229859] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11526.936790] LustreError: 392795:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11526.953700] LustreError: 392795:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11526.963714] LustreError: 392795:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11530.969309] LustreError: 392844:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11530.977049] LustreError: 392844:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11538.897446] Lustre: server umount lustre-OST0000 complete [11541.389436] LustreError: 393094:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11541.400620] LustreError: 393094:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11546.596645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.626247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11546.637324] Lustre: Skipped 2 previous similar messages [11551.715160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11551.722270] Lustre: Skipped 1 previous similar message [11553.848282] Lustre: server umount lustre-MDT0000 complete [11556.839931] LustreError: 389806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.866932] LustreError: 389806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11557.749852] LustreError: 389790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844203 with bad export cookie 1197417526151145222 [11557.756104] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.758463] LustreError: 389790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11567.268635] Lustre: server umount lustre-MDT0001 complete [11577.680176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11582.173108] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11585.106359] LustreError: 394255:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11591.050269] Lustre: 394594:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.150@tcp [11593.113720] Lustre: server umount MGS complete [11604.993310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.482736] LustreError: 395318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11605.500795] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11605.509157] Lustre: Skipped 1 previous similar message [11605.588741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11609.652875] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11615.716197] LustreError: 395318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.740424] LustreError: 395318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11618.638756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.947363] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11622.939102] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11630.995175] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11636.955133] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11638.980880] Lustre: Permanently reactivating lustre-OST0000 [11639.020135] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11639.032947] Lustre: Skipped 2 previous similar messages [11667.423172] LustreError: 397107:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11675.344216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11675.946063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11675.959043] Lustre: Skipped 1 previous similar message [11684.170593] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11693.087431] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11701.168924] Lustre: server umount lustre-OST0000 complete [11706.338849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11706.362245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.379158] Lustre: Skipped 1 previous similar message [11706.397901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11715.125972] Lustre: server umount lustre-MDT0000 complete [11718.053721] Lustre: 386621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844348/real 1788844348] req@000000005cae5b2f x1875738904070528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788844364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11718.109238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11718.141949] Lustre: Skipped 1 previous similar message [11718.150863] LustreError: 397314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.177475] LustreError: 397314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11719.166448] LustreError: 396052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844365 with bad export cookie 1197417526151146272 [11719.169210] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11719.173422] LustreError: 396052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11719.487418] Lustre: server umount lustre-MDT0001 complete [11730.632410] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [11733.302335] Key type lgssc unregistered [11733.568702] LNet: 399683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11733.574037] LNetError: 399683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11733.605519] LNet: Removed LNI 192.168.203.150@tcp [11734.627842] Key type .llcrypt unregistered [11734.629767] Key type ._llcrypt unregistered [11752.928146] Key type ._llcrypt registered [11752.939304] Key type .llcrypt registered [11753.074401] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [11768.998532] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [11770.218598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11770.308746] alg: No test for adler32 (adler32-zlib) [11771.368527] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [11771.608714] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [11773.272311] Key type lgssc registered [11774.462413] Lustre: Echo OBD driver; http://www.lustre.org/ [11813.553767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11813.601786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11814.866293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11814.916214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11814.972134] Lustre: lustre-MDT0000: new disk, initializing [11815.045225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11815.061680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11819.337294] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11829.394652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11829.536952] Lustre: 402838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11829.587771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11829.592149] Lustre: Skipped 1 previous similar message [11829.651757] Lustre: lustre-MDT0001: new disk, initializing [11829.758171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11829.812383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11829.826239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11834.030243] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11843.702337] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11849.252949] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11855.696908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11855.929656] Lustre: lustre-OST0000: new disk, initializing [11855.933237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11855.939707] Lustre: 404808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11856.006101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11861.413439] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11864.123959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11864.133978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11864.221636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11870.253354] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11875.003306] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11875.209682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11880.041950] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11880.228658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11885.030306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11885.037946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.045558] LustreError: Skipped 1 previous similar message [11885.048664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11885.061975] Lustre: Skipped 1 previous similar message [11890.150210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11890.157305] Lustre: Skipped 2 previous similar messages [11895.265830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11895.275619] Lustre: Skipped 1 previous similar message [11896.289516] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11896.587091] Lustre: server umount lustre-OST0000 complete [11900.386185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.405711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11900.413900] Lustre: Skipped 1 previous similar message [11905.505187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11905.510851] Lustre: Skipped 1 previous similar message [11906.500079] Lustre: server umount lustre-MDT0000 complete [11909.713696] LustreError: 403633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844555 with bad export cookie 5839309412404616418 [11909.723608] LustreError: 403633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11909.724700] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11909.982740] Lustre: server umount lustre-MDT0001 complete [11918.009638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11920.274079] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:16:04 (1788844564) [11928.144900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.956105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11933.591457] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11934.188828] LustreError: 407266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11934.206751] LustreError: 407266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11938.272540] LustreError: 407267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.231853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.398799] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11943.557373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11948.266918] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11956.369083] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11962.879350] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11969.480372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11969.878952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11975.565434] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [11983.233368] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11984.999673] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11987.529842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11998.699669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11998.705150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11998.711485] LustreError: Skipped 1 previous similar message [11998.726307] Lustre: Skipped 1 previous similar message [11998.730262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11998.742315] Lustre: Skipped 1 previous similar message [12003.712261] Lustre: server umount lustre-OST0000 complete [12008.931655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.009310] Lustre: server umount lustre-MDT0000 complete [12014.056050] LustreError: 407263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.074193] LustreError: 407263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12017.858023] LustreError: 407247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844663 with bad export cookie 5839309412404617307 [12017.865989] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12018.195932] Lustre: server umount lustre-MDT0001 complete [12031.204662] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12034.829116] Key type lgssc unregistered [12035.192085] LNet: 410992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12035.210482] LNetError: 410992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12035.245662] LNet: Removed LNI 192.168.203.150@tcp [12036.299145] Key type .llcrypt unregistered [12036.306460] Key type ._llcrypt unregistered [12061.064456] Key type ._llcrypt registered [12061.068674] Key type .llcrypt registered [12061.171643] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12061.503618] Key type .llcrypt unregistered [12061.505954] Key type ._llcrypt unregistered [12079.109368] Key type ._llcrypt registered [12079.112162] Key type .llcrypt registered [12079.219456] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12080.177396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12080.193889] alg: No test for adler32 (adler32-zlib) [12081.233134] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12081.484213] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12083.192125] Key type lgssc registered [12084.139913] Lustre: Echo OBD driver; http://www.lustre.org/ [12090.789048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12090.804881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12092.200380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12095.922369] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12102.632262] LustreError: 413199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.651529] LustreError: 413199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12103.511749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12103.811107] LustreError: 413200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.843034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12103.844244] LustreError: 413200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12109.282620] LustreError: 413194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.320329] LustreError: 413194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12110.621958] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12114.403068] LustreError: 413200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.497767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12117.922203] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12122.597317] LustreError: 413200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.620935] LustreError: 413200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12125.488560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12126.044165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12129.717981] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12137.953342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12137.970178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.988272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12139.466708] Lustre: server umount lustre-OST0000 complete [12146.148914] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12146.152933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.161976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12149.164581] Lustre: server umount lustre-MDT0000 complete [12151.779780] LustreError: 413199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.800740] LustreError: 413199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12152.840163] LustreError: 413181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844798 with bad export cookie 12428500392588896114 [12152.841765] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12152.861650] LustreError: 413181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12162.244511] Lustre: server umount lustre-MDT0001 complete [12173.644041] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12176.424241] Key type lgssc unregistered [12176.680611] LNet: 416347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12176.696091] LNetError: 416347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12176.718746] LNet: Removed LNI 192.168.203.150@tcp [12177.506292] Key type .llcrypt unregistered [12177.512284] Key type ._llcrypt unregistered [12199.009560] Key type ._llcrypt registered [12199.014501] Key type .llcrypt registered [12199.150831] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12200.033544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12200.088237] alg: No test for adler32 (adler32-zlib) [12201.205389] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12201.380558] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12203.039326] Key type lgssc registered [12204.182704] Lustre: Echo OBD driver; http://www.lustre.org/ [12213.568748] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12213.602163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12229.216800] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12234.337788] LustreError: 417714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [12261.151870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12268.499260] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12277.497628] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12284.855212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12285.593792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12285.676552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12291.042934] LustreError: 417739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.071101] LustreError: 417739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12291.136543] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12295.152800] LustreError: 417740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12295.164077] LustreError: 417740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12299.829275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.278206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12305.359328] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12314.406526] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12320.942808] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12330.985037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12330.985245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12330.995515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12335.563637] Lustre: server umount lustre-OST0000 complete [12341.218659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12341.218913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.245214] Lustre: Skipped 1 previous similar message [12341.263866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12341.273216] Lustre: Skipped 2 previous similar messages [12346.103051] Lustre: server umount lustre-MDT0000 complete [12346.344373] LustreError: 420531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.384642] LustreError: 420531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12350.844067] LustreError: 420241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844996 with bad export cookie 4777386638486640013 [12350.857667] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12350.876085] LustreError: 420241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12351.421987] Lustre: server umount lustre-MDT0001 complete [12365.067444] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12368.567222] Key type lgssc unregistered [12368.848653] LNet: 421282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12368.855221] LNetError: 421282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12368.870395] LNet: Removed LNI 192.168.203.150@tcp [12369.916491] Key type .llcrypt unregistered [12369.922707] Key type ._llcrypt unregistered [12395.278379] Key type ._llcrypt registered [12395.283548] Key type .llcrypt registered [12395.393343] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12396.284871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12396.455537] alg: No test for adler32 (adler32-zlib) [12397.578276] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12397.884940] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12399.551521] Key type lgssc registered [12400.643285] Lustre: Echo OBD driver; http://www.lustre.org/ [12408.198958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12408.211291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12409.653465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12414.217075] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12419.056434] LustreError: 422698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12419.079189] LustreError: 422698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12422.351072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12422.677516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12427.467604] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12434.920494] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12441.193280] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12448.202222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12448.659712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12454.829651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12455.416912] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12462.762111] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12467.565807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12467.569229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12467.597420] Lustre: Skipped 2 previous similar messages [12467.630909] Lustre: Skipped 2 previous similar messages [12469.723260] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [12472.812661] LustreError: 422698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12472.835814] LustreError: 422698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12473.034358] Lustre: server umount lustre-MDT0000 complete [12474.819906] LustreError: 422693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.329819] LustreError: 422679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.50@tcp arrived at 1788845121 with bad export cookie 12555174575320248918 [12475.341155] LustreError: 422679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12477.300451] LustreError: 422678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845123 with bad export cookie 12555174575320248029 [12477.317856] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.320556] LustreError: 422678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12477.920961] LustreError: 422692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12477.927170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.955834] LustreError: 422692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12477.978079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12479.496233] Lustre: server umount lustre-OST0000 complete [12483.046317] LustreError: 423695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12483.080228] LustreError: 423695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12487.263410] LustreError: 423427:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000314e850c x1875739979840768/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12487.440205] Lustre: server umount lustre-MDT0001 complete [12498.135475] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12500.907696] Key type lgssc unregistered [12501.331863] LNet: 426379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12501.344042] LNetError: 426379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12501.365934] LNet: Removed LNI 192.168.203.150@tcp [12502.436398] Key type .llcrypt unregistered [12502.448388] Key type ._llcrypt unregistered [12526.704671] Key type ._llcrypt registered [12526.710796] Key type .llcrypt registered [12526.845826] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12527.870634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12527.906183] alg: No test for adler32 (adler32-zlib) [12528.933087] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12529.200761] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12530.855191] Key type lgssc registered [12532.051752] Lustre: Echo OBD driver; http://www.lustre.org/ [12540.898963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12540.934944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12542.370059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12547.815644] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12552.684503] LustreError: 427794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12552.720235] LustreError: 427794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12557.794396] LustreError: 427795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12559.225647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12559.692927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12564.612756] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12573.483297] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12579.816537] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12587.225815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12587.495793] LustreError: 429617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12587.678976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12593.147123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12594.577331] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12602.574712] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12609.505854] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12609.520413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12609.526877] Lustre: Skipped 1 previous similar message [12610.509323] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [12613.662573] Lustre: server umount lustre-MDT0001 complete [12614.634865] LustreError: 427789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12614.648373] LustreError: 427789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12618.722811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12618.734374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12618.746794] Lustre: Skipped 1 previous similar message [12618.751148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12619.747596] LustreError: 427794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12624.986410] Lustre: server umount lustre-OST0000 complete [12628.907795] Lustre: server umount lustre-MDT0000 complete [12640.822402] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12643.172729] Key type lgssc unregistered [12643.491735] LNet: 431479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12643.514313] LNetError: 431479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12643.536835] LNet: Removed LNI 192.168.203.150@tcp [12644.310399] Key type .llcrypt unregistered [12644.312935] Key type ._llcrypt unregistered [12666.758462] Key type ._llcrypt registered [12666.767113] Key type .llcrypt registered [12666.888186] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [12681.104733] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12682.095656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12682.123819] alg: No test for adler32 (adler32-zlib) [12683.177413] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12683.432613] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12685.103178] Key type lgssc registered [12686.122426] Lustre: Echo OBD driver; http://www.lustre.org/ [12728.063539] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12746.855527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12746.873920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12748.250502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12748.292837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12748.392703] Lustre: lustre-MDT0000: new disk, initializing [12748.502630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12748.534177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12753.205306] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12770.162830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12770.435852] Lustre: lustre-OST0000: new disk, initializing [12770.439216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12770.445943] Lustre: Skipped 1 previous similar message [12770.451288] Lustre: 436512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12770.553126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12776.680320] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12779.134182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12779.142396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12779.178599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12786.274109] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12801.980223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12802.097584] Lustre: 435494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12802.178384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12802.312476] Lustre: lustre-MDT0001: new disk, initializing [12802.497723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12802.538307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12802.556461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12807.133076] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [12817.047241] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12817.404696] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12822.258704] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12822.510548] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12830.168682] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.50@tcp (stopping) [12831.713447] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12831.716952] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12831.737422] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12834.119871] Lustre: server umount lustre-MDT0001 complete [12841.953473] LustreError: 436969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12842.003765] LustreError: 436969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12847.077711] LustreError: 435501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12847.103643] LustreError: 435501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12852.216593] LustreError: 435505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12852.251056] LustreError: 435505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12856.289115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12856.299707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.320628] Lustre: Skipped 1 previous similar message [12856.325222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12856.338863] Lustre: Skipped 1 previous similar message [12857.327663] LustreError: 435504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.734351] Lustre: server umount lustre-OST0000 complete [12862.440060] LustreError: 435505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.039584] Lustre: server umount lustre-MDT0000 complete [12884.780067] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12888.193729] Key type lgssc unregistered [12888.497798] LNet: 440257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12888.504458] LNetError: 440257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12888.541792] LNet: Removed LNI 192.168.203.150@tcp [12889.515994] Key type .llcrypt unregistered [12889.518817] Key type ._llcrypt unregistered [12932.855549] Key type ._llcrypt registered [12932.862821] Key type .llcrypt registered [12932.969623] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [12933.249509] Key type .llcrypt unregistered [12933.251375] Key type ._llcrypt unregistered [12950.016719] Key type ._llcrypt registered [12950.024722] Key type .llcrypt registered [12950.175816] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [12951.570746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12951.730930] alg: No test for adler32 (adler32-zlib) [12952.889557] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [12953.168803] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [12954.847184] Key type lgssc registered [12956.249985] Lustre: Echo OBD driver; http://www.lustre.org/ [12988.201269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12988.256484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.571960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12989.621136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12989.790645] Lustre: lustre-MDT0000: new disk, initializing [12989.941143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12989.962375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12995.669382] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13007.872381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13008.113648] Lustre: 443628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13008.228101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13008.236568] Lustre: Skipped 1 previous similar message [13008.432487] Lustre: lustre-MDT0001: new disk, initializing [13008.580719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13008.641975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13008.660506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13015.044449] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13024.786436] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13030.845820] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13038.171856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13038.584520] Lustre: lustre-OST0000: new disk, initializing [13038.616767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13038.650907] Lustre: 445597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13038.845529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13045.205601] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13045.841464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13045.851291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13046.039687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13055.783270] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13065.184805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13065.196501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13065.225072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13066.735254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13066.736316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13066.759553] Lustre: Skipped 1 previous similar message [13066.776962] Lustre: Skipped 2 previous similar messages [13068.974680] Lustre: server umount lustre-MDT0000 complete [13073.074341] LustreError: 444433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845719 with bad export cookie 14101550804077270932 [13073.080467] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13073.083297] LustreError: 444433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13073.412159] Lustre: server umount lustre-MDT0001 complete [13087.852604] Lustre: server umount lustre-OST0000 complete [13098.334593] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:35:43 (1788845743) [13107.272648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13107.808110] LustreError: 447625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13107.838457] LustreError: 447625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13107.905978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13112.269280] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13113.313275] LustreError: 447626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13117.429917] LustreError: 447625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13120.479495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13120.918924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13125.050530] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13132.658231] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13139.254297] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13159.659564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13159.940827] Lustre: Found index 0 for lustre-OST0000, updating log [13160.171252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13166.947273] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13174.162932] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13178.608167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13183.970741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.975396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13183.976128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13183.983889] Lustre: Skipped 1 previous similar message [13185.727042] Lustre: server umount lustre-OST0000 complete [13202.919169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13202.929419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13202.943169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13202.951474] Lustre: Skipped 3 previous similar messages [13211.620719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13211.627030] Lustre: Skipped 3 previous similar messages [13216.223171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13216.408247] Lustre: server umount lustre-MDT0000 complete [13217.761639] LustreError: 447622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13217.788658] LustreError: 447622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13220.226135] LustreError: 447607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845866 with bad export cookie 14101550804077272325 [13220.233965] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13220.234461] LustreError: 447607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13220.623630] Lustre: server umount lustre-MDT0001 complete [13229.711552] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:37:54 (1788845874) [13229.729215] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [13852.635070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13853.322356] LustreError: 454043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13853.408842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13858.643931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13858.788908] LustreError: 454044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13863.906848] LustreError: 454043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13867.311468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13867.782715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13872.827792] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13880.730429] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13887.269609] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13907.624918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13907.858782] Lustre: Found index 0 for lustre-OST0000, updating log [13908.174817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13908.646062] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [13914.822801] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [13923.173821] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13925.857565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13925.881614] Lustre: Skipped 1 previous similar message [13925.894935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13925.895275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13925.914216] Lustre: Skipped 2 previous similar messages [13931.615585] Lustre: server umount lustre-OST0000 complete [13936.738314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13936.751183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13936.764839] Lustre: Skipped 1 previous similar message [13938.853211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13938.865531] Lustre: Skipped 4 previous similar messages [13941.929191] LustreError: 454038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13941.940058] LustreError: 454038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13942.081351] Lustre: server umount lustre-MDT0000 complete [13946.148860] LustreError: 456180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13946.181639] LustreError: 456180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13946.251614] LustreError: 454778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846592 with bad export cookie 14101550804077273410 [13946.254065] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13946.269307] LustreError: 454778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13946.617575] Lustre: server umount lustre-MDT0001 complete [13968.031335] LNet: 1 local NIs in recovery (showing 1): 192.168.203.150@tcp [13977.751969] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [13987.567552] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14029.885439] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:51:14 (1788846674) [14051.573346] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [14054.440647] Key type lgssc unregistered [14054.723894] LNet: 460655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14054.733016] LNetError: 460655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14169.447539] LNet: Removed LNI 192.168.203.150@tcp [14170.335674] Key type .llcrypt unregistered [14170.338634] Key type ._llcrypt unregistered [14187.992803] Key type ._llcrypt registered [14187.999542] Key type .llcrypt registered [14188.139405] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14189.669524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14189.897354] alg: No test for adler32 (adler32-zlib) [14190.975244] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [14191.236770] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [14192.991175] Key type lgssc registered [14194.516891] Lustre: Echo OBD driver; http://www.lustre.org/ [14200.133109] LNet: Added LNI 192.168.203.150@tcp100 [8/256/0/180] [14201.076240] LNet: Added LNI 192.168.203.150@tcp101 [8/256/0/180] [14202.067786] LNet: Added LNI 192.168.203.150@tcp102 [8/256/0/180] [14203.042704] LNet: Added LNI 192.168.203.150@tcp103 [8/256/0/180] [14204.026102] LNet: Added LNI 192.168.203.150@tcp104 [8/256/0/180] [14205.286647] LNet: Added LNI 192.168.203.150@tcp105 [8/256/0/180] [14206.272399] LNet: Added LNI 192.168.203.150@tcp106 [8/256/0/180] [14207.283846] LNet: Added LNI 192.168.203.150@tcp107 [8/256/0/180] [14208.418820] LNet: Added LNI 192.168.203.150@tcp108 [8/256/0/180] [14209.457389] LNet: Added LNI 192.168.203.150@tcp109 [8/256/0/180] [14210.566617] LNet: Added LNI 192.168.203.150@tcp110 [8/256/0/180] [14211.498148] LNet: Added LNI 192.168.203.150@tcp111 [8/256/0/180] [14212.404863] LNet: Added LNI 192.168.203.150@tcp112 [8/256/0/180] [14213.573293] LNet: Added LNI 192.168.203.150@tcp113 [8/256/0/180] [14214.602489] LNet: Added LNI 192.168.203.150@tcp114 [8/256/0/180] [14215.768919] LNet: Added LNI 192.168.203.150@tcp115 [8/256/0/180] [14216.845659] LNet: Added LNI 192.168.203.150@tcp116 [8/256/0/180] [14217.847616] LNet: Added LNI 192.168.203.150@tcp117 [8/256/0/180] [14218.961478] LNet: Added LNI 192.168.203.150@tcp118 [8/256/0/180] [14220.032521] LNet: Added LNI 192.168.203.150@tcp119 [8/256/0/180] [14220.915805] LNet: Added LNI 192.168.203.150@tcp120 [8/256/0/180] [14221.765597] LNet: Added LNI 192.168.203.150@tcp121 [8/256/0/180] [14222.803866] LNet: Added LNI 192.168.203.150@tcp122 [8/256/0/180] [14223.777362] LNet: Added LNI 192.168.203.150@tcp123 [8/256/0/180] [14224.636017] LNet: Added LNI 192.168.203.150@tcp124 [8/256/0/180] [14225.548755] LNet: Added LNI 192.168.203.150@tcp125 [8/256/0/180] [14226.524779] LNet: Added LNI 192.168.203.150@tcp126 [8/256/0/180] [14227.495037] LNet: Added LNI 192.168.203.150@tcp127 [8/256/0/180] [14228.284688] LNet: Added LNI 192.168.203.150@tcp128 [8/256/0/180] [14229.108948] LNet: Added LNI 192.168.203.150@tcp129 [8/256/0/180] [14230.018183] LNet: Added LNI 192.168.203.150@tcp130 [8/256/0/180] [14231.061470] LNet: Added LNI 192.168.203.150@tcp131 [8/256/0/180] [14232.174896] LNet: Added LNI 192.168.203.150@tcp132 [8/256/0/180] [14233.102295] LNet: Added LNI 192.168.203.150@tcp133 [8/256/0/180] [14234.013985] LNet: Added LNI 192.168.203.150@tcp134 [8/256/0/180] [14234.759626] LNet: Added LNI 192.168.203.150@tcp135 [8/256/0/180] [14241.577100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14241.635636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14242.723994] Lustre: 463683:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [14242.804483] LNetError: 461140:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14242.821499] LNetError: Protocol error connecting to 192.168.203.150@tcp125 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14242.832333] LNetError: Skipped 7 previous similar messages [14242.943543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14242.974476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14243.085787] Lustre: lustre-MDT0000: new disk, initializing [14243.146423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14243.174684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14243.487412] LNet: 6 local NIs in recovery (showing 6): 192.168.203.150@tcp127, 192.168.203.150@tcp128, 192.168.203.150@tcp126, 192.168.203.150@tcp129, 192.168.203.150@tcp124, 192.168.203.150@tcp125 [14243.488926] LNetError: 461143:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14243.523459] LNetError: 461143:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14243.538806] LNetError: Protocol error connecting to 192.168.203.150@tcp117 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14243.545818] LNetError: Skipped 3 previous similar messages [14247.454731] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14257.279951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14257.437626] Lustre: 463699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14257.492795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14257.496138] Lustre: Skipped 1 previous similar message [14257.567495] Lustre: lustre-MDT0001: new disk, initializing [14257.604074] Lustre: 464499:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [14257.620083] Lustre: 464499:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14257.775374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14257.809178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14257.827921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14261.799619] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14264.865315] Lustre: 464870:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [14264.880633] Lustre: 464870:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14272.062623] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14272.163287] LNetError: 463703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14272.174897] LNetError: 463703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14272.187839] LNetError: Protocol error connecting to 192.168.203.150@tcp122 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14274.210472] LNetError: 463714:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14274.220867] LNetError: 463714:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14274.231770] LNetError: Protocol error connecting to 192.168.203.150@tcp125 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14274.240974] LNetError: Skipped 1 previous similar message [14278.304267] LNetError: 461142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14278.311799] LNetError: 461142:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14278.339581] LNetError: Protocol error connecting to 192.168.203.150@tcp118 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14278.359053] LNetError: Skipped 2 previous similar messages [14278.519276] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14286.500200] LNetError: 463708:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14286.514985] LNetError: 463708:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14286.536474] LNetError: Protocol error connecting to 192.168.203.150@tcp117 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14286.560625] LNetError: Skipped 2 previous similar messages [14287.077360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14287.570987] Lustre: lustre-OST0000: new disk, initializing [14287.579188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14287.592391] Lustre: 465672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14287.717651] Lustre: 465677:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [14287.729670] Lustre: 465677:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14287.803292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14294.058713] Lustre: 465963:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.150@tcp: can't add NID 192.168.203.150@tcp130: rc = -75 [14294.084373] Lustre: 465963:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14294.173270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14294.195119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14294.404603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14295.424311] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14302.880267] LNetError: 463708:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14302.904810] LNetError: 463708:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14302.915979] LNetError: Protocol error connecting to 192.168.203.150@tcp115 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14302.925526] LNetError: Skipped 9 previous similar messages [14307.200760] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14320.113860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14320.121172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14320.139232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14320.143557] Lustre: Skipped 1 previous similar message [14320.157663] Lustre: Skipped 1 previous similar message [14323.449272] Lustre: server umount lustre-OST0000 complete [14329.312082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14329.317720] LustreError: Skipped 1 previous similar message [14329.320866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14329.342285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14330.336742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14330.358390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14330.361348] Lustre: Skipped 1 previous similar message [14333.568274] Lustre: server umount lustre-MDT0000 complete [14336.677118] LNetError: 463708:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.150 [14336.687761] LNetError: 463708:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14336.702350] LNetError: Protocol error connecting to 192.168.203.150@tcp122 at host 192.168.203.150:988: is it running a compatible version of Lustre? [14336.715735] LNetError: Skipped 9 previous similar messages [14337.309572] LustreError: 463691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846983 with bad export cookie 13985874638096491751 [14337.317255] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14337.324536] LustreError: 463691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14337.706981] Lustre: server umount lustre-MDT0001 complete [14348.421897] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [14350.942459] Key type lgssc unregistered [14351.342834] LNet: 467586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14351.354799] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14351.365488] LNet: Removed LNI 192.168.203.150@tcp [14352.418057] LNet: Removed LNI 192.168.203.150@tcp100 [14352.423947] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14352.441490] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14352.453122] LNet: Removed LNI 192.168.203.150@tcp101 [14353.503194] LNet: Removed LNI 192.168.203.150@tcp102 [14353.507567] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14353.524665] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14353.533228] LNet: Removed LNI 192.168.203.150@tcp103 [14354.591819] LNet: Removed LNI 192.168.203.150@tcp104 [14355.615663] LNet: Removed LNI 192.168.203.150@tcp105 [14355.627934] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14355.639499] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14355.650859] LNet: Removed LNI 192.168.203.150@tcp106 [14356.703696] LNet: Removed LNI 192.168.203.150@tcp107 [14357.727352] LNet: Removed LNI 192.168.203.150@tcp108 [14358.751483] LNet: Removed LNI 192.168.203.150@tcp109 [14359.775164] LNet: Removed LNI 192.168.203.150@tcp110 [14359.780613] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14359.791708] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14359.802557] LNet: Removed LNI 192.168.203.150@tcp111 [14360.863178] LNet: Removed LNI 192.168.203.150@tcp112 [14361.889978] LNet: Removed LNI 192.168.203.150@tcp113 [14362.975149] LNet: Removed LNI 192.168.203.150@tcp114 [14363.999320] LNet: Removed LNI 192.168.203.150@tcp115 [14365.023401] LNet: Removed LNI 192.168.203.150@tcp116 [14366.047158] LNet: Removed LNI 192.168.203.150@tcp117 [14367.071373] LNet: Removed LNI 192.168.203.150@tcp118 [14368.095148] LNet: Removed LNI 192.168.203.150@tcp119 [14368.097414] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14368.101464] LNetError: 467586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14368.106205] LNet: Removed LNI 192.168.203.150@tcp120 [14369.119377] LNet: Removed LNI 192.168.203.150@tcp121 [14370.143308] LNet: Removed LNI 192.168.203.150@tcp122 [14371.167569] LNet: Removed LNI 192.168.203.150@tcp123 [14372.196114] LNet: Removed LNI 192.168.203.150@tcp124 [14373.215429] LNet: Removed LNI 192.168.203.150@tcp125 [14374.239241] LNet: Removed LNI 192.168.203.150@tcp126 [14375.267750] LNet: Removed LNI 192.168.203.150@tcp127 [14376.287130] LNet: Removed LNI 192.168.203.150@tcp128 [14377.312497] LNet: Removed LNI 192.168.203.150@tcp129 [14377.318839] LNet: Removed LNI 192.168.203.150@tcp130 [14377.326300] LNet: Removed LNI 192.168.203.150@tcp131 [14377.331548] LNet: Removed LNI 192.168.203.150@tcp132 [14377.333342] LNet: Removed LNI 192.168.203.150@tcp133 [14377.339873] LNet: Removed LNI 192.168.203.150@tcp134 [14377.357768] LNet: Removed LNI 192.168.203.150@tcp135 [14378.171863] Key type .llcrypt unregistered [14378.174750] Key type ._llcrypt unregistered [14403.274262] Key type ._llcrypt registered [14403.276851] Key type .llcrypt registered [14403.420182] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [14421.439635] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14422.794893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14422.803843] alg: No test for adler32 (adler32-zlib) [14423.889389] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [14424.141147] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [14425.783168] Key type lgssc registered [14426.870452] Lustre: Echo OBD driver; http://www.lustre.org/ [14467.283195] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:58:32 (1788847112) [14484.134444] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [14487.811306] Key type lgssc unregistered [14488.237580] LNet: 471671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14488.246452] LNetError: 471671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14488.270125] LNet: Removed LNI 192.168.203.150@tcp [14489.178658] Key type .llcrypt unregistered [14489.185920] Key type ._llcrypt unregistered [14504.693729] Key type ._llcrypt registered [14504.697581] Key type .llcrypt registered [14504.797750] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14505.483512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14505.493669] alg: No test for adler32 (adler32-zlib) [14506.656415] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [14506.977633] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [14508.727390] Key type lgssc registered [14509.871380] Lustre: Echo OBD driver; http://www.lustre.org/ [14522.440235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14522.501802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14523.864176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14523.915295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14524.046830] Lustre: lustre-MDT0000: new disk, initializing [14524.178834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14524.221756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14529.025072] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14540.825295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14540.882352] Lustre: 472882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14540.933922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14540.939268] Lustre: Skipped 1 previous similar message [14541.005994] Lustre: lustre-MDT0001: new disk, initializing [14541.080604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14541.150827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14541.191782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14545.104366] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14555.726948] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14561.705491] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14568.560112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14569.011993] Lustre: lustre-OST0000: new disk, initializing [14569.019534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14569.030354] Lustre: 474850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14569.141480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14575.587278] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14578.817963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14578.835035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14578.966951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14585.408426] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14590.116885] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing lnet_if_list [14595.331663] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14595.395156] LNet: Added LNI 192.168.203.150@tcp10 [8/256/0/180] [14602.288232] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14602.348046] LNet: Added LNI 192.168.203.150@tcp11 [8/256/0/180] [14607.075281] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14607.131273] LNet: Added LNI 192.168.203.150@tcp12 [8/256/0/180] [14611.668897] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14611.704827] LNet: Added LNI 192.168.203.150@tcp13 [8/256/0/180] [14616.508706] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14616.550622] LNet: Added LNI 192.168.203.150@tcp14 [8/256/0/180] [14621.491577] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14621.532098] LNet: Added LNI 192.168.203.150@tcp15 [8/256/0/180] [14626.525282] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14626.558625] LNet: Added LNI 192.168.203.150@tcp16 [8/256/0/180] [14631.513918] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14631.547668] LNet: Added LNI 192.168.203.150@tcp17 [8/256/0/180] [14636.065114] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14636.104898] LNet: Added LNI 192.168.203.150@tcp18 [8/256/0/180] [14641.700796] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14641.754263] LNet: Added LNI 192.168.203.150@tcp19 [8/256/0/180] [14647.635372] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14647.673121] LNet: Added LNI 192.168.203.150@tcp20 [8/256/0/180] [14652.793460] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14652.826728] LNet: Added LNI 192.168.203.150@tcp21 [8/256/0/180] [14658.636944] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14658.670140] LNet: Added LNI 192.168.203.150@tcp22 [8/256/0/180] [14664.804154] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14664.841558] LNet: Added LNI 192.168.203.150@tcp23 [8/256/0/180] [14670.800099] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14670.828796] LNet: Added LNI 192.168.203.150@tcp24 [8/256/0/180] [14677.025252] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14677.064888] LNet: Added LNI 192.168.203.150@tcp25 [8/256/0/180] [14683.477247] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14683.542638] LNet: Added LNI 192.168.203.150@tcp26 [8/256/0/180] [14690.524464] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14690.554270] LNet: Added LNI 192.168.203.150@tcp27 [8/256/0/180] [14696.417064] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14696.448978] LNet: Added LNI 192.168.203.150@tcp28 [8/256/0/180] [14702.615816] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14702.652614] LNet: Added LNI 192.168.203.150@tcp29 [8/256/0/180] [14709.448380] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14709.492631] LNet: Added LNI 192.168.203.150@tcp30 [8/256/0/180] [14716.202796] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14716.232946] LNet: Added LNI 192.168.203.150@tcp31 [8/256/0/180] [14723.745283] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14723.798556] LNet: Added LNI 192.168.203.150@tcp32 [8/256/0/180] [14732.684589] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14732.724296] LNet: Added LNI 192.168.203.150@tcp33 [8/256/0/180] [14740.916673] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14740.980185] LNet: Added LNI 192.168.203.150@tcp34 [8/256/0/180] [14748.605290] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14748.636448] LNet: Added LNI 192.168.203.150@tcp35 [8/256/0/180] [14756.711081] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14756.780110] LNet: Added LNI 192.168.203.150@tcp36 [8/256/0/180] [14764.421670] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14764.487410] LNet: Added LNI 192.168.203.150@tcp37 [8/256/0/180] [14772.292915] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14772.321807] LNet: Added LNI 192.168.203.150@tcp38 [8/256/0/180] [14779.226125] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14779.274282] LNet: Added LNI 192.168.203.150@tcp39 [8/256/0/180] [14786.511340] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14786.553071] LNet: Added LNI 192.168.203.150@tcp40 [8/256/0/180] [14794.115112] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14794.157700] LNet: Added LNI 192.168.203.150@tcp41 [8/256/0/180] [14802.174795] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14802.213130] LNet: Added LNI 192.168.203.150@tcp42 [8/256/0/180] [14809.781779] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14809.818963] LNet: Added LNI 192.168.203.150@tcp43 [8/256/0/180] [14817.805410] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14817.839863] LNet: Added LNI 192.168.203.150@tcp44 [8/256/0/180] [14825.141369] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14825.175104] LNet: Added LNI 192.168.203.150@tcp45 [8/256/0/180] [14832.600593] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14832.632115] LNet: Added LNI 192.168.203.150@tcp46 [8/256/0/180] [14841.035155] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14841.069595] LNet: Added LNI 192.168.203.150@tcp47 [8/256/0/180] [14849.953550] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14849.980190] LNet: Added LNI 192.168.203.150@tcp48 [8/256/0/180] [14858.008544] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14858.031937] LNet: Added LNI 192.168.203.150@tcp49 [8/256/0/180] [14867.020776] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14867.065715] LNet: Added LNI 192.168.203.150@tcp50 [8/256/0/180] [14889.457366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14889.470546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14889.486803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14891.493092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14891.494822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14891.520787] Lustre: Skipped 1 previous similar message [14891.544049] Lustre: Skipped 2 previous similar messages [14893.190742] Lustre: server umount lustre-MDT0000 complete [14897.079930] LustreError: 472875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847543 with bad export cookie 5158928317242593220 [14897.086202] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14897.094672] LustreError: 472875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14897.678359] Lustre: server umount lustre-MDT0001 complete [14917.002189] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [14917.088373] Lustre: 472153:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847547/real 1788847547] req@00000000ef082f5e x1875742192476800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847563 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14917.113151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14918.112249] Lustre: 472152:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847548/real 1788847548] req@00000000fd6d29c3 x1875742192476928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14922.335780] Lustre: 472152:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847552/real 1788847552] req@0000000017a0835a x1875742192477056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14926.204256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14929.375285] Lustre: 472154:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788847559/real 1788847559] req@000000005597f316 x1875742192477440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788847575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14929.431532] Lustre: 472154:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14942.782855] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x479831f8aaf2d9a3 [14942.797263] Lustre: MGC192.168.203.150@tcp: Connection restored to 0@lo (at 0@lo) [14943.065383] LustreError: 474844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14943.094534] LustreError: 474844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14943.144579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14946.208199] LustreError: 474845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14946.218347] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14947.232733] LustreError: 474844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14947.648586] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14952.417065] LustreError: 474845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14956.541497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14956.953554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14962.013324] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [14962.101029] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14969.989406] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14976.287458] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14998.002495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14998.033632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14998.036215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14998.052413] Lustre: Skipped 1 previous similar message [15003.107566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15003.116594] Lustre: Skipped 1 previous similar message [15008.225353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15008.241666] Lustre: Skipped 1 previous similar message [15010.275128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15010.509686] Lustre: server umount lustre-OST0000 complete [15018.464918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15018.471266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15018.483146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15018.504772] Lustre: Skipped 2 previous similar messages [15018.515126] Lustre: Skipped 1 previous similar message [15020.687549] Lustre: server umount lustre-MDT0000 complete [15023.584574] LustreError: 499121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15023.611786] LustreError: 499121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15024.783890] LustreError: 472873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847670 with bad export cookie 5158928317242595747 [15024.786360] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15024.800372] LustreError: 472873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15025.442465] Lustre: server umount lustre-MDT0001 complete [15036.509365] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [15038.984637] Key type lgssc unregistered [15039.378063] LNet: 501921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15039.401798] LNetError: 501921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15039.410746] LNet: Removed LNI 192.168.203.150@tcp [15039.414721] LNet: Removed LNI 192.168.203.150@tcp10 [15039.418911] LNet: Removed LNI 192.168.203.150@tcp11 [15039.423292] LNet: Removed LNI 192.168.203.150@tcp12 [15039.430866] LNet: Removed LNI 192.168.203.150@tcp13 [15039.436974] LNet: Removed LNI 192.168.203.150@tcp14 [15039.444405] LNet: Removed LNI 192.168.203.150@tcp15 [15039.448802] LNet: Removed LNI 192.168.203.150@tcp16 [15039.452626] LNet: Removed LNI 192.168.203.150@tcp17 [15039.465492] LNet: Removed LNI 192.168.203.150@tcp18 [15039.467583] LNet: Removed LNI 192.168.203.150@tcp19 [15039.473745] LNet: Removed LNI 192.168.203.150@tcp20 [15039.477880] LNet: Removed LNI 192.168.203.150@tcp21 [15039.485811] LNet: Removed LNI 192.168.203.150@tcp22 [15039.493454] LNet: Removed LNI 192.168.203.150@tcp23 [15039.500933] LNet: Removed LNI 192.168.203.150@tcp24 [15039.507331] LNet: Removed LNI 192.168.203.150@tcp25 [15039.512451] LNet: Removed LNI 192.168.203.150@tcp26 [15039.517506] LNet: Removed LNI 192.168.203.150@tcp27 [15039.522286] LNet: Removed LNI 192.168.203.150@tcp28 [15039.527922] LNet: Removed LNI 192.168.203.150@tcp29 [15039.533512] LNet: Removed LNI 192.168.203.150@tcp30 [15039.535516] LNet: Removed LNI 192.168.203.150@tcp31 [15039.546905] LNet: Removed LNI 192.168.203.150@tcp32 [15039.549152] LNet: Removed LNI 192.168.203.150@tcp33 [15039.553058] LNet: Removed LNI 192.168.203.150@tcp34 [15039.555930] LNet: Removed LNI 192.168.203.150@tcp35 [15039.563364] LNet: Removed LNI 192.168.203.150@tcp36 [15039.569285] LNet: Removed LNI 192.168.203.150@tcp37 [15039.573386] LNet: Removed LNI 192.168.203.150@tcp38 [15039.577225] LNet: Removed LNI 192.168.203.150@tcp39 [15039.582490] LNet: Removed LNI 192.168.203.150@tcp40 [15039.585477] LNet: Removed LNI 192.168.203.150@tcp41 [15039.591484] LNet: Removed LNI 192.168.203.150@tcp42 [15039.594853] LNet: Removed LNI 192.168.203.150@tcp43 [15039.598976] LNet: Removed LNI 192.168.203.150@tcp44 [15039.603066] LNet: Removed LNI 192.168.203.150@tcp45 [15039.609058] LNet: Removed LNI 192.168.203.150@tcp46 [15039.613165] LNet: Removed LNI 192.168.203.150@tcp47 [15039.617488] LNet: Removed LNI 192.168.203.150@tcp48 [15039.621597] LNet: Removed LNI 192.168.203.150@tcp49 [15039.633212] LNet: Removed LNI 192.168.203.150@tcp50 [15040.405254] Key type .llcrypt unregistered [15040.409524] Key type ._llcrypt unregistered [15062.281718] Key type ._llcrypt registered [15062.285809] Key type .llcrypt registered [15062.400726] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [15080.849952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [15082.059864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15082.101851] alg: No test for adler32 (adler32-zlib) [15083.201384] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [15083.559564] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [15085.279327] Key type lgssc registered [15086.384684] Lustre: Echo OBD driver; http://www.lustre.org/ [15122.520134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15122.532963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15123.805714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15123.843764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15123.913699] Lustre: lustre-MDT0000: new disk, initializing [15124.010390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15124.043122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15130.250632] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15144.209046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15144.467800] Lustre: 505114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15144.637125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15144.647658] Lustre: Skipped 1 previous similar message [15144.750214] Lustre: lustre-MDT0001: new disk, initializing [15144.860693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15144.953619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15144.997397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15150.311128] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15160.029952] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15166.626022] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15175.436047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15175.736617] Lustre: *** cfs_fail_loc=90f, val=0*** [15175.877531] Lustre: lustre-OST0000: new disk, initializing [15175.883453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15175.893416] Lustre: 507132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15176.054209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15182.829955] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15183.405648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15183.410851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15183.542080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15194.920867] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15204.322050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15204.330024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15204.337231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15204.341436] LustreError: Skipped 1 previous similar message [15204.363786] Lustre: Skipped 1 previous similar message [15209.460425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15209.464561] Lustre: Skipped 2 previous similar messages [15214.570598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15214.580507] Lustre: Skipped 1 previous similar message [15217.119149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15217.396473] Lustre: server umount lustre-OST0000 complete [15221.728336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15221.737333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15221.757202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15224.814498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15227.785680] Lustre: server umount lustre-MDT0000 complete [15232.740611] LustreError: 505107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847878 with bad export cookie 17498873056769753520 [15232.763449] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15232.767815] LustreError: 505107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15233.567159] Lustre: server umount lustre-MDT0001 complete [15248.335890] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [15251.089130] Key type lgssc unregistered [15251.414609] LNet: 509040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15251.431520] LNetError: 509040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15252.454964] LNet: Removed LNI 192.168.203.150@tcp [15253.282346] Key type .llcrypt unregistered [15253.285088] Key type ._llcrypt unregistered [15271.251402] Key type ._llcrypt registered [15271.253230] Key type .llcrypt registered [15271.320723] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [15272.196108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15272.234641] alg: No test for adler32 (adler32-zlib) [15273.315821] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [15273.626108] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [15275.287197] Key type lgssc registered [15276.857221] Lustre: Echo OBD driver; http://www.lustre.org/ [15285.012572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15285.039944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15286.584483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15291.801273] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15295.977115] LustreError: 510311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15296.004825] LustreError: 510311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15301.098306] LustreError: 510312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15301.433929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15302.099663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15307.569127] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15317.747790] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15327.091826] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15336.431385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15336.739635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15343.862435] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15353.206654] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15373.958255] loop0: detected capacity change from 0 to 204800000 [15374.349365] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15374.379183] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15374.407297] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15384.597216] loop0: detected capacity change from 0 to 204800000 [15384.848127] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15384.865811] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15384.897637] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15391.183940] loop0: detected capacity change from 0 to 204800000 [15397.378556] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15397.542723] Lustre: fs15246-OST0000: new disk, initializing [15397.546606] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15397.556185] Lustre: 513928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15397.603832] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15405.454519] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15416.982968] loop1: detected capacity change from 0 to 204800000 [15423.009668] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15423.165731] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15423.218638] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15423.333357] Lustre: fs15246-MDT0000: new disk, initializing [15423.546842] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15423.582559] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15423.720541] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15424.800230] Lustre: 515051:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000005f258fb x1875742995569024/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788848086 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15428.646783] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15433.975579] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15433.995784] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15434.032544] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15452.652670] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15452.679531] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15457.778131] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15458.357417] Lustre: server umount fs15246-MDT0000 complete [15462.891825] LustreError: 510308:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15468.010141] LustreError: 510308:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15468.929358] Lustre: server umount fs15246-OST0000 complete [15483.367715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15483.379447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15483.400589] Lustre: Skipped 1 previous similar message [15483.410433] Lustre: Skipped 1 previous similar message [15488.494052] LustreError: 512134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15488.515714] LustreError: 512134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15488.690391] Lustre: server umount lustre-OST0000 complete [15493.601335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15493.603142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15493.639607] Lustre: Skipped 1 previous similar message [15493.658635] Lustre: Skipped 1 previous similar message [15498.720291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15498.731741] Lustre: Skipped 1 previous similar message [15499.630482] Lustre: server umount lustre-MDT0000 complete [15503.848609] LustreError: 511260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15503.879887] LustreError: 511260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15504.321471] LustreError: 510292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848150 with bad export cookie 438124459814501108 [15504.329537] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15504.332701] LustreError: 510292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15504.681838] Lustre: server umount lustre-MDT0001 complete [15517.285185] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [15520.406207] Key type lgssc unregistered [15520.823814] LNet: 518146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15520.837488] LNetError: 518146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15520.867900] LNet: Removed LNI 192.168.203.150@tcp [15522.154496] Key type .llcrypt unregistered [15522.157484] Key type ._llcrypt unregistered [15543.041226] Key type ._llcrypt registered [15543.048301] Key type .llcrypt registered [15543.191337] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [15544.431547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15544.508517] alg: No test for adler32 (adler32-zlib) [15545.715808] Lustre: Lustre: Build Version: 2.17.57_106_g45f4cc0 [15546.148385] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [15547.864807] Key type lgssc registered [15549.388149] Lustre: Echo OBD driver; http://www.lustre.org/ [15592.842880] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [15605.027745] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [15645.030480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15645.076553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15646.451117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15646.517866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15646.628314] Lustre: lustre-MDT0000: new disk, initializing [15646.728288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15646.757688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15652.210798] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15664.917949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15665.068518] Lustre: 522615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15665.152634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15665.156479] Lustre: Skipped 1 previous similar message [15665.250278] Lustre: lustre-MDT0001: new disk, initializing [15665.410990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15665.474809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15665.487893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15671.381227] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15682.983851] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15691.102129] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15700.866062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15701.197556] Lustre: lustre-OST0000: new disk, initializing [15701.205167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15701.214686] Lustre: 524585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15701.328225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15708.918416] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [15710.785921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15710.795579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15710.897520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15718.982075] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15724.546950] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15724.797247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15729.607717] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15729.890985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15736.805758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15736.813437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15736.816862] LustreError: Skipped 1 previous similar message [15736.818150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15736.867756] Lustre: Skipped 1 previous similar message [15741.920115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15741.937680] Lustre: Skipped 2 previous similar messages [15746.015469] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15746.447768] Lustre: server umount lustre-OST0000 complete [15752.160717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15752.162753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15752.183406] Lustre: Skipped 1 previous similar message [15752.209615] Lustre: Skipped 1 previous similar message [15756.341797] Lustre: server umount lustre-MDT0000 complete [15760.054855] LustreError: 522608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848406 with bad export cookie 8348941349123928833 [15760.059087] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15760.063982] LustreError: 522608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15760.456285] Lustre: server umount lustre-MDT0001 complete [15787.514177] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [15799.147850] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [15835.664847] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15584 sec ========= 02:21:20 (1788848480) [15837.731578] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:21:22 (1788848482) === [15841.590130] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:21:26 (1788848486) === [15890.744385] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [15894.178746] Key type lgssc unregistered [15894.481713] LNet: 531231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15894.490553] LNetError: 531231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15894.520798] LNet: Removed LNI 192.168.203.150@tcp [15895.583427] Key type .llcrypt unregistered [15895.587536] Key type ._llcrypt unregistered