[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 774604947 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001103] APIC: Switch to symmetric I/O mode setup [ 0.003396] x2apic enabled [ 0.004021] Switched APIC routing to physical x2apic. [ 0.006016] kvm-guest: setup PV IPIs [ 0.009478] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012021] pid_max: default: 32768 minimum: 301 [ 0.013172] LSM: Security Framework initializing [ 0.014097] Yama: becoming mindful. [ 0.015073] SELinux: Initializing. [ 0.016108] *** VALIDATE selinux *** [ 0.027241] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035247] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036204] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038483] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040014] *** VALIDATE tmpfs *** [ 0.042121] *** VALIDATE proc *** [ 0.043301] *** VALIDATE cgroup *** [ 0.044015] *** VALIDATE cgroup2 *** [ 0.045331] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.048020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.049047] Spectre V2 : User space: Vulnerable [ 0.051020] Speculative Store Bypass: Vulnerable [ 0.055630] debug: unmapping init [mem 0xffffffffb0a59000-0xffffffffb0a60fff] [ 0.057339] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.059115] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.060047] ... version: 2 [ 0.061026] ... bit width: 48 [ 0.062020] ... generic registers: 4 [ 0.063021] ... value mask: 0000ffffffffffff [ 0.064020] ... max period: 00007fffffffffff [ 0.065023] ... fixed-purpose events: 3 [ 0.066019] ... event mask: 000000070000000f [ 0.067500] rcu: Hierarchical SRCU implementation. [ 0.070319] smp: Bringing up secondary CPUs ... [ 0.071803] x86: Booting SMP configuration: [ 0.072049] .... node #0, CPUs: #1 #2 #3 [ 0.081240] smp: Brought up 1 node, 4 CPUs [ 0.083046] smpboot: Max logical packages: 1 [ 0.084014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.118078] node 0 deferred pages initialised in 31ms [ 0.121500] devtmpfs: initialized [ 0.122236] x86/mm: Memory block size: 128MB [ 0.127173] gcov: version magic: 0x41383552 [ 0.128324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.129155] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.130478] pinctrl core: initialized pinctrl subsystem [ 0.131000] [ 0.132069] ************************************************************* [ 0.134011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136013] ** ** [ 0.139012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141012] ** ** [ 0.143012] ** This means that this kernel is built to expose internal ** [ 0.147015] ** IOMMU data structures, which may compromise security on ** [ 0.149013] ** your system. ** [ 0.152012] ** ** [ 0.154011] ** If you see this message and you are not debugging the ** [ 0.156013] ** kernel, report this immediately to your vendor! ** [ 0.158012] ** ** [ 0.159011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161019] ************************************************************* [ 0.163915] NET: Registered protocol family 16 [ 0.167017] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172143] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175111] cpuidle: using governor menu [ 0.177476] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180549] PCI: Using configuration type 1 for base access [ 0.182426] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193149] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195058] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198164] cryptd: max_cpu_qlen set to 1000 [ 0.200382] ACPI: Added _OSI(Module Device) [ 0.202016] ACPI: Added _OSI(Processor Device) [ 0.204014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212207] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221007] ACPI: Interpreter enabled [ 0.222054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223013] ACPI: Using IOAPIC for interrupt routing [ 0.226137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229610] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241833] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253714] acpiphp: Slot [2] registered [ 0.255249] acpiphp: Slot [5] registered [ 0.257089] acpiphp: Slot [6] registered [ 0.258094] acpiphp: Slot [7] registered [ 0.259000] acpiphp: Slot [8] registered [ 0.259000] acpiphp: Slot [9] registered [ 0.261082] acpiphp: Slot [10] registered [ 0.262082] acpiphp: Slot [3] registered [ 0.264111] acpiphp: Slot [4] registered [ 0.265080] acpiphp: Slot [11] registered [ 0.266084] acpiphp: Slot [12] registered [ 0.267121] acpiphp: Slot [13] registered [ 0.269093] acpiphp: Slot [14] registered [ 0.270084] acpiphp: Slot [15] registered [ 0.271096] acpiphp: Slot [16] registered [ 0.272217] acpiphp: Slot [17] registered [ 0.274080] acpiphp: Slot [18] registered [ 0.275122] acpiphp: Slot [19] registered [ 0.277095] acpiphp: Slot [20] registered [ 0.278075] acpiphp: Slot [21] registered [ 0.279078] acpiphp: Slot [22] registered [ 0.281153] acpiphp: Slot [23] registered [ 0.282077] acpiphp: Slot [24] registered [ 0.283076] acpiphp: Slot [25] registered [ 0.284330] acpiphp: Slot [26] registered [ 0.286093] acpiphp: Slot [27] registered [ 0.287143] acpiphp: Slot [28] registered [ 0.289093] acpiphp: Slot [29] registered [ 0.290000] acpiphp: Slot [30] registered [ 0.290000] acpiphp: Slot [31] registered [ 0.291074] PCI host bridge to bus 0000:00 [ 0.293023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.305243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.308215] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311468] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322672] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326248] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336522] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341104] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.343822] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374140] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415267] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.443017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453862] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.460016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.466017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.482021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.493252] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.504021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.524022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.535646] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.540016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.541000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.546000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.546000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.551019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.557029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.573020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.585683] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.590147] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.592345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.594326] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.596354] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.601310] iommu: Default domain type: Passthrough [ 0.602000] SCSI subsystem initialized [ 0.602000] ACPI: bus type USB registered [ 0.604105] usbcore: registered new interface driver usbfs [ 0.605056] usbcore: registered new interface driver hub [ 0.607093] usbcore: registered new device driver usb [ 0.608146] pps_core: LinuxPPS API ver. 1 registered [ 0.609011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.611057] PTP clock support registered [ 0.613087] EDAC MC: Ver: 3.0.0 [ 0.614779] PCI: Using ACPI for IRQ routing [ 0.615859] NetLabel: Initializing [ 0.616007] NetLabel: domain hash size = 128 [ 0.617010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.618139] NetLabel: unlabeled traffic allowed by default [ 0.619339] vgaarb: loaded [ 0.621700] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.623012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.629515] clocksource: Switched to clocksource kvm-clock [ 0.796874] VFS: Disk quotas dquot_6.6.0 [ 0.798588] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801628] *** VALIDATE ramfs *** [ 0.802949] *** VALIDATE hugetlbfs *** [ 0.804783] pnp: PnP ACPI init [ 0.807573] pnp: PnP ACPI: found 6 devices [ 0.842270] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.846609] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.848700] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850870] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.856043] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.859091] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.861387] NET: Registered protocol family 2 [ 0.863363] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.868049] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.871435] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.875897] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879151] TCP: Hash tables configured (established 65536 bind 65536) [ 0.881634] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.884097] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.886340] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888925] NET: Registered protocol family 1 [ 0.892171] RPC: Registered named UNIX socket transport module. [ 0.894774] RPC: Registered udp transport module. [ 0.896805] RPC: Registered tcp transport module. [ 0.899083] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.901313] NET: Registered protocol family 44 [ 0.902391] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.904316] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.905367] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.906796] PCI: CLS 0 bytes, default 64 [ 0.908187] Unpacking initramfs... [ 3.616537] debug: unmapping init [mem 0xffff8b50fcc54000-0xffff8b50fffbffff] [ 3.623883] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.627348] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.632265] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.216099] Initialise system trusted keyrings [ 6.229850] Key type blacklist registered [ 6.233850] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.246064] zbud: loaded [ 6.249557] *** VALIDATE nfs *** [ 6.250633] *** VALIDATE nfs4 *** [ 6.252373] pstore: using deflate compression [ 6.277261] Platform Keyring initialized [ 6.813812] NET: Registered protocol family 38 [ 6.881580] Key type asymmetric registered [ 6.883027] Asymmetric key parser 'x509' registered [ 6.885025] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.889328] io scheduler mq-deadline registered [ 6.892766] io scheduler kyber registered [ 6.896105] io scheduler bfq registered [ 6.899787] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.904991] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.910604] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.916698] ACPI: Power Button [PWRF] [ 6.929971] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.942906] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.000947] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.062977] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.347680] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.405797] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.436238] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.441837] Non-volatile memory driver v1.3 [ 7.444625] Linux agpgart interface v0.103 [ 7.491504] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 7.495415] vda: detected capacity change from 0 to 74321920 [ 7.537721] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.541362] vdb: detected capacity change from 0 to 1073741824 [ 7.559628] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.562843] vdc: detected capacity change from 0 to 2621440000 [ 7.588528] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.593383] vdd: detected capacity change from 0 to 2621440000 [ 7.615298] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.619516] vde: detected capacity change from 0 to 4294967296 [ 7.761684] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.774018] vdf: detected capacity change from 0 to 4294967296 [ 7.809811] libphy: Fixed MDIO Bus: probed [ 7.907942] usbcore: registered new interface driver usbserial_generic [ 7.911363] usbserial: USB Serial support registered for generic [ 7.915369] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.921851] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.929333] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.932805] mousedev: PS/2 mouse device common for all mice [ 7.938938] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.942121] rtc_cmos 00:05: RTC can wake from S4 [ 7.952304] rtc_cmos 00:05: registered as rtc0 [ 7.959350] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.961097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.965824] intel_pstate: CPU model not supported [ 7.977258] hid: raw HID events driver (C) Jiri Kosina [ 7.983020] usbcore: registered new interface driver usbhid [ 7.993592] usbhid: USB HID core driver [ 7.999719] drop_monitor: Initializing network drop monitor service [ 8.007614] Initializing XFRM netlink socket [ 8.008350] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.012793] NET: Registered protocol family 10 [ 8.021834] Segment Routing with IPv6 [ 8.025974] NET: Registered protocol family 17 [ 8.029259] mpls_gso: MPLS GSO support [ 8.038667] RAS: Correctable Errors collector initialized. [ 8.045847] AVX version of gcm_enc/dec engaged. [ 8.052647] AES CTR mode by8 optimization enabled [ 8.366927] sched_clock: Marking stable (8366903239, 0)->(9521362042, -1154458803) [ 8.373242] registered taskstats version 1 [ 8.374931] Loading compiled-in X.509 certificates [ 8.377772] zswap: loaded using pool lzo/zbud [ 8.410860] Key type big_key registered [ 8.424271] Key type encrypted registered [ 8.425399] ima: No TPM chip found, activating TPM-bypass! [ 8.427255] ima: Allocated hash algorithm: sha1 [ 8.429221] ima: No architecture policies found [ 8.430783] evm: Initialising EVM extended attributes: [ 8.432104] evm: security.selinux [ 8.432916] evm: security.ima [ 8.433886] evm: security.capability [ 8.435013] evm: HMAC attrs: 0x1 [ 8.436992] rtc_cmos 00:05: setting system clock to 2026-07-03 09:32:07 UTC (1783071127) [ 8.443722] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 8.446149] debug: unmapping init [mem 0xffffffffb0782000-0xffffffffb0a58fff] [ 8.453406] Write protecting the kernel read-only data: 28672k [ 8.457782] debug: unmapping init [mem 0xffffffffaee03000-0xffffffffaeffffff] [ 8.459936] debug: unmapping init [mem 0xffffffffaf714000-0xffffffffaf7fffff] [ 8.498943] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.509537] systemd[1]: Detected virtualization kvm. [ 8.511960] systemd[1]: Detected architecture x86-64. [ 8.513837] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.544577] systemd[1]: No hostname configured. [ 8.547215] systemd[1]: Set hostname to . [ 8.550314] random: systemd: uninitialized urandom read (16 bytes read) [ 8.555552] systemd[1]: Initializing machine ID from random generator. [ 8.845945] random: ln: uninitialized urandom read (6 bytes read) [ 9.066901] random: systemd: uninitialized urandom read (16 bytes read) [ 9.069935] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.080880] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.091493] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.951945] device-mapper: uevent: version 1.0.3 [ 12.962471] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.803911] random: fast init done [ 16.015606] virtio_net virtio0 ens2: renamed from eth0 [ 17.328447] scsi host0: ata_piix [ 17.436363] scsi host1: ata_piix [ 17.441485] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.453568] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.943646] random: crng init done [ 21.946939] random: 7 urandom warning(s) missed due to ratelimiting [ 25.293594] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 27.228196] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.431502] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.757425] SELinux: Disabled at runtime. [ 33.989563] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.030110] systemd[1]: Detected virtualization kvm. [ 34.035369] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.451458] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.460167] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.479127] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.491379] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.516197] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.573236] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.600719] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 37.543486] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 38.553312] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 40.149845] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 40.458884] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 41.725718] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.899352] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 49.520196] hrtimer: interrupt took 7318209 ns [ **] A start job is running for Configur…only root support (13s / no limit)[ 50.368283] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit)[ 51.902547] NFS: Registering the id_resolver key type [ 51.907535] Key type id_resolver registered [ 51.910426] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [** ] A start job is running for Rebuild …amic Linker Cache (23s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg152-server login: [ 115.502461] libcfs: loading out-of-tree module taints kernel. [ 115.555711] Key type ._llcrypt registered [ 115.557968] Key type .llcrypt registered [ 115.660462] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 132.104295] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 133.379411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 133.389413] alg: No test for adler32 (adler32-zlib) [ 134.815726] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 135.417398] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 137.192532] Key type lgssc registered [ 138.804237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 153.748971] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.370716] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 205.138401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 205.160977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.464334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.527474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.646995] Lustre: lustre-MDT0000: new disk, initializing [ 206.739168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.762531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 211.685282] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.186194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.316849] Lustre: 6513:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 225.338964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 225.342556] Lustre: Skipped 1 previous similar message [ 225.413176] Lustre: lustre-MDT0001: new disk, initializing [ 225.484431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.510421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 225.525935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 229.952773] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.334563] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.993293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.347633] Lustre: lustre-OST0000: new disk, initializing [ 245.355818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.362476] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.436925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.428260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.440208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 251.535106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.837969] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.590990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.702346] Lustre: lustre-OST0001: new disk, initializing [ 266.713254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.718109] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 266.766851] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.799863] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.519296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 276.539127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 276.618212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.244433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 292.958031] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 301.132621] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.301660] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing yml_node [ 311.309199] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 314.063574] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 316.755870] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 318.595217] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 3 05:37:15 EDT 2026 [ 338.874138] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 340.386228] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.370076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.372640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.386663] Lustre: Skipped 3 previous similar messages [ 358.409614] Lustre: Skipped 3 previous similar messages [ 362.563488] Lustre: server umount lustre-MDT0000 complete [ 366.955024] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071486 with bad export cookie 13982579963870110875 [ 366.956248] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.972667] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 368.610605] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.614505] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.617894] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.640488] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 368.676034] Lustre: Skipped 1 previous similar message [ 371.616262] LustreError: 10097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.617943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.636039] LustreError: 10097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 371.659800] Lustre: Skipped 2 previous similar messages [ 373.496675] Lustre: server umount lustre-MDT0001 complete [ 377.866349] Lustre: server umount lustre-OST0000 complete [ 382.077072] Lustre: server umount lustre-OST0001 complete [ 390.154668] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 401.812835] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 441.749804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.014784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.037643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.118102] Lustre: lustre-MDT0000: new disk, initializing [ 442.176468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.190797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.662728] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 457.532899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.768878] Lustre: 15075:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 457.872497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 457.880862] Lustre: Skipped 1 previous similar message [ 458.034114] Lustre: lustre-MDT0001: new disk, initializing [ 458.185929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.257915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.263671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 463.510197] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 473.978945] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 480.422539] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 487.794776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.201330] Lustre: lustre-OST0000: new disk, initializing [ 488.209416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.220930] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.400576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.770885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 489.782344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 489.946649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 495.813179] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 505.923104] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 511.252913] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.744889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 516.764269] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 516.988077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 520.676802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.678690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 520.700353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.717241] Lustre: Skipped 1 previous similar message [ 525.207386] Lustre: server umount lustre-OST0000 complete [ 529.888490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 529.899241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.919432] Lustre: Skipped 1 previous similar message [ 529.928140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.130627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.138523] Lustre: Skipped 5 previous similar messages [ 543.199100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 543.477105] Lustre: server umount lustre-MDT0000 complete [ 545.253341] LustreError: 15088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.280631] LustreError: 15088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 547.663139] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071666 with bad export cookie 13982579963870112401 [ 547.668332] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.674193] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 548.252424] Lustre: server umount lustre-MDT0001 complete [ 552.527943] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:41:09 (1783071669) [ 555.036032] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 556.790273] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:41:14 (1783071674) [ 563.883164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.164917] LustreError: 19456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.214660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.655837] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 569.332606] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.525452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.206339] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 587.936812] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 593.612668] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 600.657830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 607.683499] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 615.176978] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 622.561975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 622.570670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.586505] Lustre: Skipped 1 previous similar message [ 622.599592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 624.619076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.594740] Lustre: server umount lustre-OST0000 complete [ 633.315844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 633.327658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.358102] Lustre: Skipped 1 previous similar message [ 638.517543] Lustre: server umount lustre-MDT0000 complete [ 639.973139] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.995419] LustreError: 19457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 642.350889] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783071761 with bad export cookie 13982579963870113290 [ 642.359786] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 642.373316] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.717857] Lustre: server umount lustre-MDT0001 complete [ 654.489122] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 657.642624] Key type lgssc unregistered [ 658.035844] LNet: 22988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.043554] LNetError: 22988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.064162] LNet: Removed LNI 192.168.201.152@tcp [ 659.323301] Key type .llcrypt unregistered [ 659.326436] Key type ._llcrypt unregistered [ 689.836468] Key type ._llcrypt registered [ 689.837950] Key type .llcrypt registered [ 690.067161] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 691.845615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.000351] alg: No test for adler32 (adler32-zlib) [ 693.459732] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 694.071603] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 696.033868] Key type lgssc registered [ 697.749777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 709.748845] loop: module loaded [ 710.510519] loop0: detected capacity change from 0 to 204800000 [ 710.861169] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.879903] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 711.442320] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.004099] loop0: detected capacity change from 0 to 204800000 [ 719.315542] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.325814] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.344772] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.212109] loop0: detected capacity change from 0 to 204800000 [ 730.638532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 730.660238] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.975982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 732.134503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 732.253557] Lustre: lustre-MDT0000: new disk, initializing [ 732.405315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.448672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 737.015687] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 746.542232] loop1: detected capacity change from 0 to 204800000 [ 753.652883] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 754.188666] Lustre: lustre-OST07c6: new disk, initializing [ 754.195755] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 754.210540] Lustre: Skipped 1 previous similar message [ 754.216578] Lustre: 26180:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 754.359417] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 762.353793] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 762.610800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 762.635375] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 762.676550] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 768.716733] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 768.730509] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 768.739895] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 768.745963] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 768.751917] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 768.756665] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 788.296142] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 788.325967] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 788.352348] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.373840] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 788.398534] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 788.409501] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.404693] Lustre: 26787:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 793.410705] Lustre: 26787:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.414549] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.422588] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.430374] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.439827] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.453598] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.459722] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.476649] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 793.481983] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.488592] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.502265] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.704549] Lustre: 26787:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 796.714489] Lustre: 26787:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 796.719575] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 796.723705] Lustre: 26787:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.727737] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 796.733398] Lustre: 26787:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.738513] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 796.743310] Lustre: 26787:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.755637] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 796.759988] Lustre: 26787:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.770574] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 796.776361] Lustre: 26787:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.627064] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 801.633738] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 801.640455] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 801.648100] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.655636] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 801.660655] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.665769] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 801.671651] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.677900] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 801.684477] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 801.689044] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 801.694606] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.841777] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 812.853344] Lustre: 25166:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 812.865540] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 812.874731] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 812.889734] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.898502] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 812.907898] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 812.920882] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 812.935288] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 812.942747] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 812.952788] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.963745] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 860.131924] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.169977] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 865.256693] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 866.142069] Lustre: server umount lustre-OST07c6 complete [ 878.373726] Lustre: server umount lustre-MDT0000 complete [ 900.332267] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 902.956973] Key type lgssc unregistered [ 903.230080] LNet: 28737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.244730] LNetError: 28737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.273928] LNet: Removed LNI 192.168.201.152@tcp [ 904.188368] Key type .llcrypt unregistered [ 904.192826] Key type ._llcrypt unregistered [ 923.998742] Key type ._llcrypt registered [ 924.004872] Key type .llcrypt registered [ 924.182783] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 925.924949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.998521] alg: No test for adler32 (adler32-zlib) [ 927.121564] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 927.403121] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 929.151678] Key type lgssc registered [ 930.200423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 937.804124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 937.826657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.342907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.590971] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 949.737882] LustreError: 30009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.753444] LustreError: 30009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 952.514771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.792172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.177909] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 964.666715] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 971.461992] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 978.566284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 978.985789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.186263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 986.307533] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 994.349611] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1013.217591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.221302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.230885] Lustre: Skipped 1 previous similar message [ 1018.355753] Lustre: server umount lustre-OST0000 complete [ 1023.465123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.496062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.513043] Lustre: Skipped 2 previous similar messages [ 1028.577811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.606033] Lustre: Skipped 1 previous similar message [ 1028.937214] Lustre: server umount lustre-MDT0000 complete [ 1033.697379] LustreError: 30010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.723672] LustreError: 30010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1033.769372] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072152 with bad export cookie 8056155514251080935 [ 1033.784950] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1033.785975] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.696322] Lustre: server umount lustre-MDT0001 complete [ 1047.392417] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1049.989841] Key type lgssc unregistered [ 1050.298735] LNet: 33748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.307132] LNetError: 33748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.321711] LNet: Removed LNI 192.168.201.152@tcp [ 1051.467570] Key type .llcrypt unregistered [ 1051.470544] Key type ._llcrypt unregistered [ 1077.161812] Key type ._llcrypt registered [ 1077.168127] Key type .llcrypt registered [ 1077.299723] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1078.745748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.829858] alg: No test for adler32 (adler32-zlib) [ 1079.980336] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1080.217442] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1081.958906] Key type lgssc registered [ 1083.673148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.025144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1094.067633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.699742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.546339] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1105.898625] LustreError: 35022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.925869] LustreError: 35022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1110.506949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.823459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.508706] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1124.318066] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.218418] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1140.289513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1140.708971] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1140.894425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.400948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1148.760664] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1158.293785] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1170.913181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.916831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.920041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.943970] Lustre: Skipped 1 previous similar message [ 1170.956469] Lustre: Skipped 1 previous similar message [ 1175.097198] Lustre: server umount lustre-OST0000 complete [ 1181.163904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.180797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.279572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.298458] Lustre: Skipped 2 previous similar messages [ 1186.490604] Lustre: server umount lustre-MDT0000 complete [ 1191.067745] LustreError: 36865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072310 with bad export cookie 2120290646087816075 [ 1191.077944] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.084728] LustreError: 36865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1191.416797] Lustre: server umount lustre-MDT0001 complete [ 1202.986312] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1205.496979] Key type lgssc unregistered [ 1205.960628] LNet: 38556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.975821] LNetError: 38556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.001305] LNet: Removed LNI 192.168.201.152@tcp [ 1206.741093] Key type .llcrypt unregistered [ 1206.743898] Key type ._llcrypt unregistered [ 1232.622627] Key type ._llcrypt registered [ 1232.624937] Key type .llcrypt registered [ 1232.721027] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1233.987505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1234.024344] alg: No test for adler32 (adler32-zlib) [ 1235.178469] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1235.495550] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1237.151139] Key type lgssc registered [ 1238.588657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1248.646986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1248.680384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.396641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.867318] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1260.515927] LustreError: 39976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.535073] LustreError: 39976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1261.570765] LustreError: 39977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.673602] LustreError: 39976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.627580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.161167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1274.127363] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1282.781967] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1288.762564] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1296.417294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.804066] LustreError: 41798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.013489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.528523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1304.138945] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1312.980821] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1320.874902] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1324.007724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.012090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.052752] Lustre: Skipped 2 previous similar messages [ 1324.061651] Lustre: Skipped 1 previous similar message [ 1325.957324] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1326.827975] Lustre: server umount lustre-MDT0000 complete [ 1329.138057] LustreError: 40705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.152876] LustreError: 40705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1331.660223] LustreError: 39958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072450 with bad export cookie 7051674610528801360 [ 1331.662457] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.671395] LustreError: 39958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1334.242976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.260147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1338.379481] Lustre: server umount lustre-MDT0001 complete [ 1360.682082] Lustre: server umount lustre-OST0000 complete [ 1380.442512] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1384.074442] Key type lgssc unregistered [ 1384.377066] LNet: 43851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.394751] LNetError: 43851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.453590] LNet: Removed LNI 192.168.201.152@tcp [ 1387.097470] Key type .llcrypt unregistered [ 1387.098965] Key type ._llcrypt unregistered [ 1414.940784] Key type ._llcrypt registered [ 1414.942747] Key type .llcrypt registered [ 1415.037906] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1416.378657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.429446] alg: No test for adler32 (adler32-zlib) [ 1417.458621] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1417.652122] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1419.287705] Key type lgssc registered [ 1420.245506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1427.966796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1427.986847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.461214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.290246] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1439.714540] LustreError: 45267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.742596] LustreError: 45267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1443.840584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1444.423530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1450.446315] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1458.687076] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1465.748303] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1473.927106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1474.036212] LustreError: 47089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.309575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.660707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1481.049099] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1489.838615] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.995606] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1500.132180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.144624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.159722] LustreError: Skipped 1 previous similar message [ 1500.164425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1500.171713] Lustre: Skipped 1 previous similar message [ 1503.050561] Lustre: server umount lustre-OST0000 complete [ 1525.729276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1525.744150] Lustre: Skipped 1 previous similar message [ 1525.764268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.772872] Lustre: Skipped 1 previous similar message [ 1528.754291] Lustre: server umount lustre-MDT0000 complete [ 1530.857405] LustreError: 45264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.877124] LustreError: 45264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1532.478123] LustreError: 45247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072651 with bad export cookie 17470178645438076599 [ 1532.481445] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.492823] LustreError: 45247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1538.792989] Lustre: server umount lustre-MDT0001 complete [ 1548.732996] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1551.256450] Key type lgssc unregistered [ 1551.470537] LNet: 48979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.477340] LNetError: 48979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.492201] LNet: Removed LNI 192.168.201.152@tcp [ 1552.416232] Key type .llcrypt unregistered [ 1552.423844] Key type ._llcrypt unregistered [ 1573.168815] Key type ._llcrypt registered [ 1573.170175] Key type .llcrypt registered [ 1573.263830] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1574.288236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.371932] alg: No test for adler32 (adler32-zlib) [ 1575.434180] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1575.610298] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1577.199168] Key type lgssc registered [ 1578.188406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1584.926507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1584.957620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.338625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.088867] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1596.639262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.900034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.452648] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1606.387781] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1610.489624] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1615.091611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.372794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.749229] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1625.169815] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1632.831485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1639.300988] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1642.022605] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1653.736450] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-03 5h59m30s [ 1655.016548] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1658.337257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1658.346572] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.361625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.367090] Lustre: Skipped 2 previous similar messages [ 1658.376391] Lustre: Skipped 2 previous similar messages [ 1659.433729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.437574] Lustre: Skipped 2 previous similar messages [ 1661.337205] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1663.186232] Lustre: server umount lustre-MDT0000 complete [ 1665.518753] LustreError: 50400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.527536] LustreError: 50400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1665.601374] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1666.434119] LustreError: 50395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.676430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.771517] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.917978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.965320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1671.975358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1674.214938] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1676.827420] Lustre: DEBUG MARKER: Wait for df (36507) ... [ 1677.962846] Lustre: DEBUG MARKER: done [ 1681.378885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.385044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.400261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.406688] Lustre: Skipped 1 previous similar message [ 1682.401658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.837894] Lustre: server umount lustre-OST0000 complete [ 1692.642067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1692.642326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.653453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1692.669342] Lustre: Skipped 1 previous similar message [ 1692.674051] Lustre: Skipped 2 previous similar messages [ 1695.891287] Lustre: server umount lustre-MDT0000 complete [ 1698.825290] LustreError: 52240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072817 with bad export cookie 18125126058128240407 [ 1698.828740] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1698.836899] LustreError: 52240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1699.085879] Lustre: server umount lustre-MDT0001 complete [ 1706.381107] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1708.354292] Key type lgssc unregistered [ 1708.565074] LNet: 55343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.571561] LNetError: 55343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.604895] LNet: Removed LNI 192.168.201.152@tcp [ 1710.188722] Key type .llcrypt unregistered [ 1710.190184] Key type ._llcrypt unregistered [ 1737.436273] Key type ._llcrypt registered [ 1737.438915] Key type .llcrypt registered [ 1737.516312] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1738.330276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1738.346935] alg: No test for adler32 (adler32-zlib) [ 1739.323370] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1739.477153] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1741.127884] Key type lgssc registered [ 1741.853527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1747.536648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1747.555437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.733875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1748.747790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1748.855692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.123610] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1758.040550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.106118] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1758.126569] Lustre: 57542:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1758.337620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1761.377603] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1766.684360] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1770.817909] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1776.196029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.325784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1776.454381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.741584] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1782.900340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1786.485528] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1791.619512] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-03 6h01m49s [ 1792.710356] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1797.078663] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1798.687728] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1798.732995] LustreError: 57550:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1819.065361] Lustre: lustre-MDT0000: Client c77c2e4f-2c34-4c3b-b6c8-8dd52413d4a8 (at 192.168.201.52@tcp) reconnecting [ 1838.791113] LustreError: 57550:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1840.505824] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1842.250975] Lustre: DEBUG MARKER: done [ 1849.312360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.321655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.340351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.824643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.837538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.864142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1852.495142] Lustre: server umount lustre-OST0000 complete [ 1855.460069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1855.466145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.479610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.577112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.579522] Lustre: Skipped 4 previous similar messages [ 1861.331373] Lustre: server umount lustre-MDT0000 complete [ 1863.868179] LustreError: 57533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783072982 with bad export cookie 8377336498124558699 [ 1863.870033] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1863.872958] LustreError: 57533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1864.061490] Lustre: server umount lustre-MDT0001 complete [ 1870.647554] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1872.428765] Key type lgssc unregistered [ 1872.615208] LNet: 61861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.621909] LNetError: 61861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.634987] LNet: Removed LNI 192.168.201.152@tcp [ 1873.135083] Key type .llcrypt unregistered [ 1873.138657] Key type ._llcrypt unregistered [ 1907.097324] Key type ._llcrypt registered [ 1907.101120] Key type .llcrypt registered [ 1907.167741] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1908.156688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.173760] alg: No test for adler32 (adler32-zlib) [ 1909.152379] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 1909.386352] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1911.063200] Key type lgssc registered [ 1911.869650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1914.284120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1914.304632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.568939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.580249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1915.679148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.338221] Lustre: Failing over lustre-MDT0000 [ 1916.498719] Lustre: server umount lustre-MDT0000 complete [ 1921.218157] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:03:58 (1783073038) [ 1925.808904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.157589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.090224] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1935.111622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.171269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1935.191288] Lustre: 64638:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1935.355339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1938.029348] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1942.898888] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1946.437504] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1950.886173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1951.063572] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1951.217782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.489350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1955.800946] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1960.689697] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1964.739388] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1971.170414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.177186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.186559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.193880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.199171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.926587] Lustre: server umount lustre-MDT0000 complete [ 1977.646350] LustreError: 64630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073096 with bad export cookie 15664991167310564053 [ 1977.647034] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.656503] LustreError: 64630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1977.898854] Lustre: server umount lustre-MDT0001 complete [ 1980.310633] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1984.168760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1990.269514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.815180] Lustre: 63560:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783073101/real 1783073101] req@00000000ab85b2f4 x1869687583444224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783073117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1998.830810] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.836240] Lustre: Skipped 1 previous similar message [ 2001.955170] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd9653a78512d93b8 [ 2001.973572] Lustre: MGC192.168.201.152@tcp: Connection restored to 0@lo (at 0@lo) [ 2002.179885] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.191359] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2002.230476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.284621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2003.233891] LustreError: 66485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.260219] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783073106/real 1783073106] req@000000005cc60b9a x1869687583444352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783073122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2004.738942] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2006.244994] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.289060] LustreError: 68404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.295291] LustreError: 68404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2008.527169] Lustre: 63563:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783073111/real 1783073111] req@00000000b9d78459 x1869687583444608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783073127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.478586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.702853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.261526] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2013.791156] Lustre: 63561:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783073116/real 1783073116] req@00000000a0d426d5 x1869687583444736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783073132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2015.730687] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2018.188561] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2021.622648] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2025.953952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.956995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2025.963821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.972350] Lustre: Skipped 3 previous similar messages [ 2030.983068] Lustre: server umount lustre-MDT0000 complete [ 2032.096872] LustreError: 66485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.113248] LustreError: 66485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2033.583886] LustreError: 65373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073152 with bad export cookie 15664991167310566328 [ 2033.586696] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.594880] LustreError: 65373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2033.755956] Lustre: server umount lustre-MDT0001 complete [ 2037.889119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2043.411516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.599222] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783073156/real 1783073156] req@00000000cb68b27d x1869687583468032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783073172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2053.618173] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.628623] Lustre: Skipped 1 previous similar message [ 2058.783794] LustreError: 63559:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003cac8800 x1869687583469056/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2058.989244] LustreError: 66484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.040473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.077601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2061.317272] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2062.117165] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2064.159159] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783073167/real 1783073167] req@0000000042f39c87 x1869687583468672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783073183 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2064.172319] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2066.147704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.740754] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2071.531641] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2073.042851] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2076.148264] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2081.762400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2081.763973] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.766634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.772485] Lustre: Skipped 1 previous similar message [ 2081.779277] Lustre: Skipped 5 previous similar messages [ 2085.820784] Lustre: server umount lustre-MDT0000 complete [ 2086.881302] LustreError: 71087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.890241] LustreError: 71087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2087.692835] LustreError: 64630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073206 with bad export cookie 15664991167310567714 [ 2087.694812] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.699164] LustreError: 64630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2087.830785] Lustre: server umount lustre-MDT0001 complete [ 2091.126761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2092.898191] Lustre: DEBUG MARKER: files compared the same [ 2104.393248] Lustre: server umount lustre-OST0000 complete [ 2114.029807] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2115.609430] Key type lgssc unregistered [ 2115.792567] LNet: 74355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.800165] LNetError: 74355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.812442] LNet: Removed LNI 192.168.201.152@tcp [ 2116.288529] Key type .llcrypt unregistered [ 2116.290129] Key type ._llcrypt unregistered [ 2129.896416] Key type ._llcrypt registered [ 2129.898403] Key type .llcrypt registered [ 2129.963216] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2130.639058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2130.666714] alg: No test for adler32 (adler32-zlib) [ 2131.578767] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2131.717502] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2133.319167] Key type lgssc registered [ 2133.913943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2138.364773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2138.384613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.631926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.672043] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2146.215268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.391205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2149.297218] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2153.330098] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2156.075651] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2160.064205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.252641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.413607] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2164.339898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2167.249908] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.991995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.992210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.992659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2177.004671] Lustre: Skipped 1 previous similar message [ 2178.336054] Lustre: server umount lustre-OST0000 complete [ 2182.111666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.112274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.117492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.122252] Lustre: Skipped 1 previous similar message [ 2182.127785] Lustre: Skipped 2 previous similar messages [ 2182.307755] Lustre: server umount lustre-MDT0000 complete [ 2184.181920] LustreError: 75731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073303 with bad export cookie 10708173217036348735 [ 2184.185882] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.187194] LustreError: 75731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2184.344338] Lustre: server umount lustre-MDT0001 complete [ 2189.472280] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2190.786431] Key type lgssc unregistered [ 2190.956666] LNet: 79276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.962878] LNetError: 79276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.975086] LNet: Removed LNI 192.168.201.152@tcp [ 2192.353081] Key type .llcrypt unregistered [ 2192.354418] Key type ._llcrypt unregistered [ 2205.294963] Key type ._llcrypt registered [ 2205.296376] Key type .llcrypt registered [ 2205.359080] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2205.882829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.977377] alg: No test for adler32 (adler32-zlib) [ 2206.923218] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2207.053204] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2208.663187] Key type lgssc registered [ 2209.412794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.207472] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.219791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2229.727431] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2234.847625] LustreError: 80645:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2261.535333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.069867] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2269.757389] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2275.178879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.262876] Lustre: *** cfs_fail_loc=706, val=0*** [ 2275.560637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.618134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2278.436538] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2281.955881] LustreError: 80671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.968883] LustreError: 80671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2283.770538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.998751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2286.597986] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2290.892893] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2294.408921] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2296.289356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.302159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.319892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.361168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.361183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.385350] Lustre: Skipped 1 previous similar message [ 2301.849596] Lustre: server umount lustre-OST0000 complete [ 2304.480682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.486882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.492897] Lustre: Skipped 1 previous similar message [ 2309.604540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.609118] Lustre: Skipped 1 previous similar message [ 2314.721729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.727395] Lustre: Skipped 1 previous similar message [ 2318.303166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.411966] Lustre: server umount lustre-MDT0000 complete [ 2319.843283] LustreError: 81729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.857251] LustreError: 81729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2320.548562] LustreError: 80651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073439 with bad export cookie 5570579344349899311 [ 2320.555872] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2320.556852] LustreError: 80651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2320.751272] Lustre: server umount lustre-MDT0001 complete [ 2327.266396] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2328.978304] Key type lgssc unregistered [ 2329.182307] LNet: 84257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.194537] LNetError: 84257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.210266] LNet: Removed LNI 192.168.201.152@tcp [ 2329.765166] Key type .llcrypt unregistered [ 2329.767204] Key type ._llcrypt unregistered [ 2343.722285] Key type ._llcrypt registered [ 2343.724434] Key type .llcrypt registered [ 2343.767359] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2344.314184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.399719] alg: No test for adler32 (adler32-zlib) [ 2345.332546] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2345.517713] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2347.143297] Key type lgssc registered [ 2347.770736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2352.053693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2352.065548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.458717] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2360.022967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.321935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.164841] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2365.411068] LustreError: 86337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.428497] LustreError: 86337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2369.211893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2369.471088] LustreError: 86335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.488354] LustreError: 86335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2369.498651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.541368] LustreError: 86337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.551025] LustreError: 86337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2373.324371] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2375.650519] LustreError: 86341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.672204] LustreError: 86341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2378.656592] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.671128] Lustre: MGS: Client 24757979-1f4d-4c8f-a2a3-766cde030dcc (at 0@lo) reconnecting [ 2378.692295] LustreError: 86351:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2378.820283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.884970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2381.401698] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2394.082623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.085526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.092810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2394.098126] Lustre: Skipped 1 previous similar message [ 2397.705776] Lustre: server umount lustre-OST0000 complete [ 2404.322432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.322774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2404.328733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.338524] Lustre: Skipped 2 previous similar messages [ 2406.047687] Lustre: server umount lustre-MDT0000 complete [ 2408.468434] Lustre: server umount lustre-MDT0001 complete [ 2410.774298] Lustre: server umount MGS complete [ 2416.704176] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:12:14 (1783073534) [ 2432.743642] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 2437.718848] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2456.362246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.442177] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2463.488755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.601560] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2463.627490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2463.670425] Lustre: lustre-MDT0001: new disk, initializing [ 2463.707641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.724628] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2463.734159] Lustre: 93496:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2463.738772] LustreError: 93496:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2465.557282] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2470.702117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.821827] Lustre: lustre-OST0000: new disk, initializing [ 2470.823757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2470.826743] Lustre: 94259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2470.860618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.501317] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2473.825408] Lustre: 94485:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000fb930163 x1869688040624768/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783073608 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2477.923131] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2480.929459] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.935811] Lustre: MGS: Client fb97f216-7edf-4902-b872-4309093c4dff (at 0@lo) reconnecting [ 2480.970810] Lustre: 92764:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2480.983115] LustreError: 93481:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2480.987671] LustreError: 93481:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2480.999372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2481.039902] Lustre: lustre-MDT0000: new disk, initializing [ 2481.106960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.121390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2482.143985] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2482.152141] Lustre: 95331:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001c24a3bb x1869688040630912/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783073617 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2482.908471] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2491.231772] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2494.437742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2494.443847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2496.490669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2496.490669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2496.497970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2520.032086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2520.035365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.040684] Lustre: Skipped 1 previous similar message [ 2520.042688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.083983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2522.084455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.087638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.094802] Lustre: Skipped 1 previous similar message [ 2527.201195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.204339] Lustre: Skipped 1 previous similar message [ 2532.319156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2532.394761] Lustre: server umount lustre-OST0000 complete [ 2537.440888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2537.445620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.448927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.452547] Lustre: Skipped 1 previous similar message [ 2537.459139] Lustre: Skipped 3 previous similar messages [ 2540.439815] Lustre: server umount lustre-MDT0000 complete [ 2548.349666] Lustre: server umount lustre-MDT0001 complete [ 2549.915262] Lustre: server umount MGS complete [ 2554.806289] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:14:32 (1783073672) [ 2561.912367] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2563.177256] Key type lgssc unregistered [ 2563.330638] LNet: 97828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.334484] LNetError: 97828:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.342635] LNet: Removed LNI 192.168.201.152@tcp [ 2563.738570] Key type .llcrypt unregistered [ 2563.741527] Key type ._llcrypt unregistered [ 2572.131825] Key type ._llcrypt registered [ 2572.133186] Key type .llcrypt registered [ 2572.179685] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2572.697468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2572.712448] alg: No test for adler32 (adler32-zlib) [ 2573.599883] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2573.709398] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2575.319512] Key type lgssc registered [ 2575.774374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.480874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2579.490573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.726527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.758524] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2586.983154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.167297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.914240] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2591.723511] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2594.223359] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2597.350680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.413841] LustreError: 100845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.422491] LustreError: 100845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2597.527672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.256875] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2604.032925] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2605.032720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2613.215961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2613.218895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.223948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.268460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.271648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.278099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.281660] Lustre: Skipped 1 previous similar message [ 2618.499198] Lustre: server umount lustre-OST0000 complete [ 2620.385727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.393021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.505385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.507651] Lustre: Skipped 2 previous similar messages [ 2626.467567] Lustre: server umount lustre-MDT0000 complete [ 2628.081711] LustreError: 100869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073747 with bad export cookie 13484698422430535487 [ 2628.085611] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.089983] LustreError: 100869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2628.264035] Lustre: server umount lustre-MDT0001 complete [ 2633.008469] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2634.367165] Key type lgssc unregistered [ 2634.493524] LNet: 102559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.496813] LNetError: 102559:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2634.509533] LNet: Removed LNI 192.168.201.152@tcp [ 2634.941829] Key type .llcrypt unregistered [ 2634.942850] Key type ._llcrypt unregistered [ 2642.784184] Key type ._llcrypt registered [ 2642.785781] Key type .llcrypt registered [ 2642.827422] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2643.261834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2643.282787] alg: No test for adler32 (adler32-zlib) [ 2644.169055] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2644.275720] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2645.871122] Key type lgssc registered [ 2646.350450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.293488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.294997] LustreError: 103621:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2649.299233] Lustre: Skipped 1 previous similar message [ 2649.305581] LustreError: 103620:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2649.308972] LustreError: 103621:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2649.311494] LustreError: 103621:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2649.317053] LustreError: 103621:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2649.317753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.556539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.142982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.267347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.287283] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2658.548370] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing lsmod [ 2659.798246] LustreError: 104826:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2659.798339] LustreError: 104829:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2659.805515] LustreError: 104826:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2659.810472] LustreError: 104826:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2659.816117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.818052] LustreError: 104826:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2660.016392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.032038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2664.292433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.296023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.296979] Lustre: Skipped 2 previous similar messages [ 2664.304665] Lustre: Skipped 2 previous similar messages [ 2667.387814] Lustre: server umount lustre-MDT0000 complete [ 2669.113382] LustreError: 105015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073788 with bad export cookie 5961908069571909125 [ 2669.116909] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.124706] LustreError: 105015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2669.536461] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.540232] LustreError: 103656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.545914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2669.553473] LustreError: 103656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2670.431989] LustreError: 104090:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000078c72b4b x1869688354130048/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2670.509797] Lustre: server umount lustre-MDT0001 complete [ 2672.512197] Lustre: server umount lustre-OST0000 complete [ 2677.039909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.246894] LustreError: 105893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.286565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.104819] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2682.336039] LustreError: 105894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.048524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.236616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.960028] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2688.299783] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2690.751915] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2693.621748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.775997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.110300] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2699.102971] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2701.994130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2712.546741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2712.551881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.558550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2713.569792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.458704] Lustre: server umount lustre-OST0000 complete [ 2718.083846] Lustre: server umount lustre-MDT0000 complete [ 2718.688656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.689699] LustreError: 105894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.694446] Lustre: Skipped 2 previous similar messages [ 2719.746376] LustreError: 105873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073838 with bad export cookie 5961908069571910042 [ 2719.751298] LustreError: 105873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2719.751434] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.883535] Lustre: server umount lustre-MDT0001 complete [ 2724.291942] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2725.399870] Key type lgssc unregistered [ 2725.543704] LNet: 109440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2725.552115] LNetError: 109440:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2725.561626] LNet: Removed LNI 192.168.201.152@tcp [ 2725.906374] Key type .llcrypt unregistered [ 2725.907647] Key type ._llcrypt unregistered [ 2737.427982] Key type ._llcrypt registered [ 2737.429249] Key type .llcrypt registered [ 2737.476900] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2737.921963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2737.958336] alg: No test for adler32 (adler32-zlib) [ 2738.844505] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2738.950753] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2740.544268] Key type lgssc registered [ 2741.014891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.233320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2744.240249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.440848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.254413] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2750.940358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.123436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2752.842577] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2755.723583] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2757.823746] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2760.280869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2760.404118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.436159] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2765.215946] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2770.665837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2772.714280] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2773.739986] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2776.034463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.035914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.038593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2776.043466] Lustre: Skipped 1 previous similar message [ 2777.132220] Lustre: server umount lustre-OST0000 complete [ 2780.761915] Lustre: server umount lustre-MDT0000 complete [ 2781.153099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.159288] LustreError: 110847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.166805] LustreError: 110847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2782.198378] LustreError: 113439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073901 with bad export cookie 7213289937920799900 [ 2782.201447] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2782.202682] LustreError: 113439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2782.315261] Lustre: server umount lustre-MDT0001 complete [ 2786.431527] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2787.544801] Key type lgssc unregistered [ 2787.681395] LNet: 114479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.684310] LNetError: 114479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.693286] LNet: Removed LNI 192.168.201.152@tcp [ 2788.032728] Key type .llcrypt unregistered [ 2788.034098] Key type ._llcrypt unregistered [ 2795.617753] Key type ._llcrypt registered [ 2795.619248] Key type .llcrypt registered [ 2795.666898] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2796.024282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.071326] alg: No test for adler32 (adler32-zlib) [ 2796.950231] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2797.052587] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2798.647384] Key type lgssc registered [ 2799.093594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2802.148180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2802.156537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.311128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.781991] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2808.031373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.178306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.654689] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2812.213627] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2814.257139] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2816.750250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.886508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.141949] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2821.919665] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.136098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2838.497130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.498696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.499399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2838.502378] Lustre: Skipped 1 previous similar message [ 2841.236154] Lustre: server umount lustre-OST0000 complete [ 2843.617173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.619195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.622546] Lustre: Skipped 1 previous similar message [ 2843.624563] Lustre: Skipped 1 previous similar message [ 2848.736307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.738347] Lustre: Skipped 2 previous similar messages [ 2848.968971] Lustre: server umount lustre-MDT0000 complete [ 2850.402379] LustreError: 115658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783073969 with bad export cookie 2243990638259249724 [ 2850.404141] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.407620] LustreError: 115658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2850.528419] Lustre: server umount lustre-MDT0001 complete [ 2854.570806] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2855.686758] Key type lgssc unregistered [ 2855.829394] LNet: 119229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2855.832135] LNetError: 119229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2855.839290] LNet: Removed LNI 192.168.201.152@tcp [ 2856.195596] Key type .llcrypt unregistered [ 2856.197060] Key type ._llcrypt unregistered [ 2866.392838] Key type ._llcrypt registered [ 2866.394622] Key type .llcrypt registered [ 2866.435437] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2866.768305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2866.786749] alg: No test for adler32 (adler32-zlib) [ 2867.650938] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2867.751652] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2869.351135] Key type lgssc registered [ 2869.791919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2872.917367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2872.924572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.080527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.561849] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2878.974945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.134663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2880.754898] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2883.860344] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2886.093462] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2888.899229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.062855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.458203] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2894.934822] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2899.308090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 2903.159907] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2904.944754] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2904.950454] Lustre: Skipped 2 previous similar messages [ 2907.157865] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2907.161737] Lustre: Skipped 2 previous similar messages [ 2909.198441] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2909.214989] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2916.203991] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2916.207932] Lustre: Skipped 2 previous similar messages [ 2917.775141] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.52@tcp 192.168.0.[2,10]@tcp [ 2923.739518] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.52/24@tcp in log lustre-MDT0000 [ 2923.742823] Lustre: Skipped 2 previous similar messages [ 2925.306311] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.52/24@tcp [ 2925.308848] Lustre: Skipped 1 previous similar message [ 2930.144786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2930.146970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.151180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2930.158085] Lustre: Skipped 1 previous similar message [ 2930.163748] Lustre: Skipped 1 previous similar message [ 2935.266921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2935.269508] Lustre: Skipped 1 previous similar message [ 2936.229669] Lustre: server umount lustre-OST0000 complete [ 2940.385771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.388556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2940.394175] Lustre: Skipped 1 previous similar message [ 2940.399522] Lustre: Skipped 1 previous similar message [ 2945.505080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.507428] Lustre: Skipped 1 previous similar message [ 2950.625938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.632198] Lustre: Skipped 1 previous similar message [ 2952.159146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2952.245283] Lustre: server umount lustre-MDT0000 complete [ 2954.063711] LustreError: 123952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074073 with bad export cookie 10773696442373074138 [ 2954.065983] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2954.073379] LustreError: 123952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2954.204450] Lustre: server umount lustre-MDT0001 complete [ 2958.400408] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2959.507922] Key type lgssc unregistered [ 2959.650397] LNet: 125363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2959.653912] LNetError: 125363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2959.664462] LNet: Removed LNI 192.168.201.152@tcp [ 2960.015270] Key type .llcrypt unregistered [ 2960.016542] Key type ._llcrypt unregistered [ 2972.051284] Key type ._llcrypt registered [ 2972.052669] Key type .llcrypt registered [ 2972.099875] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2972.564468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.572340] alg: No test for adler32 (adler32-zlib) [ 2973.443591] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 2973.536554] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2975.127204] Key type lgssc registered [ 2975.555144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2978.653289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2978.661575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.829541] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2979.834901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.839192] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.52/24@tcp [ 2981.382679] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2985.146384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.313622] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2985.324504] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.52/24@tcp [ 2985.339141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2987.208915] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2990.264523] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2992.553980] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2995.490749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.639130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.913176] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3001.056293] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3004.913414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3010.019046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.019927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3010.020140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.023816] Lustre: Skipped 1 previous similar message [ 3011.738891] Lustre: server umount lustre-OST0000 complete [ 3015.136878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.137710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.141435] Lustre: Skipped 1 previous similar message [ 3015.143675] Lustre: Skipped 1 previous similar message [ 3015.252376] Lustre: server umount lustre-MDT0000 complete [ 3016.817667] LustreError: 126883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074135 with bad export cookie 11776231165075823321 [ 3016.820142] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.824528] LustreError: 126883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3016.942546] Lustre: server umount lustre-MDT0001 complete [ 3021.159278] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3022.502097] Key type lgssc unregistered [ 3022.684652] LNet: 130485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3022.689788] LNetError: 130485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3022.721244] LNet: Removed LNI 192.168.201.152@tcp [ 3023.119973] Key type .llcrypt unregistered [ 3023.121593] Key type ._llcrypt unregistered [ 3034.338644] Key type ._llcrypt registered [ 3034.339619] Key type .llcrypt registered [ 3034.386228] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3041.044413] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3041.377130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3041.384761] alg: No test for adler32 (adler32-zlib) [ 3042.255567] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3042.353269] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3043.935139] Key type lgssc registered [ 3044.351220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3057.743280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3057.749776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.837160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3058.847815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3058.883947] Lustre: lustre-MDT0000: new disk, initializing [ 3058.908315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.915691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3060.388798] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3065.099451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.151370] Lustre: 133856:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3065.178170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3065.180270] Lustre: Skipped 1 previous similar message [ 3065.218688] Lustre: lustre-MDT0001: new disk, initializing [ 3065.254285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3065.268405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3065.273055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3066.708348] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3070.688397] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3072.827191] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3075.383747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3075.491332] Lustre: lustre-OST0000: new disk, initializing [ 3075.493579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3075.496970] Lustre: 135820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3075.531886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.665257] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3081.746802] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3083.419723] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3085.820290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3085.824313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3085.863513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3086.574625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3088.176054] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3088.261744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3090.914686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3090.919441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.920745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3090.924833] Lustre: Skipped 1 previous similar message [ 3090.929617] Lustre: Skipped 1 previous similar message [ 3095.062960] Lustre: server umount lustre-OST0000 complete [ 3101.156942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.160971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3101.165422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.842096] Lustre: server umount lustre-MDT0000 complete [ 3104.496416] LustreError: 133849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074223 with bad export cookie 11593569409580514651 [ 3104.499240] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.501558] LustreError: 133849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3104.632891] Lustre: server umount lustre-MDT0001 complete [ 3108.411500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.686530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.345313] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3113.952309] LustreError: 138058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.959667] LustreError: 138058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3113.969516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.143094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3115.677035] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3118.349123] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.408602] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.332172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.476435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.680990] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3128.304228] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3129.855283] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3134.018750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3135.434224] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3135.512677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3139.120423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.183353] Lustre: lustre-OST0001: new disk, initializing [ 3139.185387] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3139.188157] Lustre: 141453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3141.388931] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3144.711218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3144.714804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3144.742202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3144.955897] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3145.039261] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3146.503612] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3146.584193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3148.654546] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91c38342e000.ost_server_uuid 50 [ 3149.233073] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91c38342e000.ost_server_uuid in FULL state after 0 sec [ 3154.914535] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.914575] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.917788] LustreError: Skipped 1 previous similar message [ 3154.923718] Lustre: Skipped 1 previous similar message [ 3154.926227] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3154.930586] Lustre: Skipped 2 previous similar messages [ 3157.500685] Lustre: server umount lustre-OST0001 complete [ 3159.520245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3159.523073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.528242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.035951] LustreError: 139879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.043113] LustreError: 139879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3165.153901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.155342] LustreError: 139878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.157595] Lustre: Skipped 3 previous similar messages [ 3170.277251] LustreError: 139877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.282402] LustreError: 139877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3173.343105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3173.403420] Lustre: server umount lustre-OST0000 complete [ 3175.392863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.394508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.398213] Lustre: Skipped 2 previous similar messages [ 3175.400555] Lustre: Skipped 2 previous similar messages [ 3180.824681] Lustre: server umount lustre-MDT0000 complete [ 3182.064182] LustreError: 138037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074301 with bad export cookie 11593569409580515519 [ 3182.066116] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3182.068819] LustreError: 138037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3182.183030] Lustre: server umount lustre-MDT0001 complete [ 3188.987941] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3190.027991] Key type lgssc unregistered [ 3190.158392] LNet: 144451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3190.161643] LNetError: 144451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3190.171332] LNet: Removed LNI 192.168.201.152@tcp [ 3190.481876] Key type .llcrypt unregistered [ 3190.483368] Key type ._llcrypt unregistered [ 3204.771095] Key type ._llcrypt registered [ 3204.772370] Key type .llcrypt registered [ 3204.812308] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3211.197415] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3211.602928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3211.610439] alg: No test for adler32 (adler32-zlib) [ 3212.503702] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3212.606583] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3214.191155] Key type lgssc registered [ 3214.610542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3227.496627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3227.502670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.581285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3228.590121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3228.624053] Lustre: lustre-MDT0000: new disk, initializing [ 3228.651826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.658450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3230.022515] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3234.401532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.446299] Lustre: 148575:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3234.471996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3234.475275] Lustre: Skipped 1 previous similar message [ 3234.512831] Lustre: lustre-MDT0001: new disk, initializing [ 3234.551955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3234.565788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3234.569689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3235.994074] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3239.994465] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3242.020380] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3244.504524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3244.610748] Lustre: lustre-OST0000: new disk, initializing [ 3244.612983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3244.615888] Lustre: 150543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3244.649846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.709842] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3250.172791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3250.176647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3250.215753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3250.787401] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3253.846279] Lustre: Failing over lustre-OST0000 [ 3253.910242] Lustre: server umount lustre-OST0000 complete [ 3266.997085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.132138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.137040] Lustre: Skipped 1 previous similar message [ 3267.141322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.148654] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3267.247579] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3269.258850] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3270.614506] Lustre: Failing over lustre-MDT0000 [ 3270.625769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3270.628035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.632942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.718351] Lustre: server umount lustre-MDT0000 complete [ 3271.647288] Lustre: 146805:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783074374/real 1783074374] req@0000000037fad46d x1869688949728896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783074390 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3272.163898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.168333] Lustre: Skipped 1 previous similar message [ 3276.255159] Lustre: 146807:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783074379/real 1783074379] req@00000000d83ad0c7 x1869688949730560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783074395 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3276.667699] LustreError: 148583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.674371] LustreError: 148583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3277.280178] LustreError: 148587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.285735] LustreError: 148587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3281.311138] Lustre: 146804:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783074384/real 1783074384] req@000000008a4ffd54 x1869688949731712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783074400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3281.787349] LustreError: 152451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.846613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.894467] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.899549] Lustre: 148579:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3283.968556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3283.985385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.006046] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3285.355462] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3286.908204] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3289.064041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3289.070768] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3289.092055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3294.179021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.179698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.180613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.189147] Lustre: Skipped 1 previous similar message [ 3296.213315] Lustre: server umount lustre-OST0000 complete [ 3299.296683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3299.297087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.297563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.297567] Lustre: Skipped 1 previous similar message [ 3299.299354] LustreError: Skipped 1 previous similar message [ 3299.303364] Lustre: Skipped 1 previous similar message [ 3303.839769] Lustre: server umount lustre-MDT0000 complete [ 3304.417218] LustreError: 148586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.422072] LustreError: 148586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3305.068164] LustreError: 149371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074424 with bad export cookie 11037698227668106748 [ 3305.070410] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.072472] LustreError: 149371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.201607] Lustre: server umount lustre-MDT0001 complete [ 3309.200278] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3310.243433] Key type lgssc unregistered [ 3310.373095] LNet: 154192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3310.377345] LNetError: 154192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3310.384324] LNet: Removed LNI 192.168.201.152@tcp [ 3310.725570] Key type .llcrypt unregistered [ 3310.727055] Key type ._llcrypt unregistered [ 3331.504957] Key type ._llcrypt registered [ 3331.506061] Key type .llcrypt registered [ 3331.541906] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3331.878911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.891290] alg: No test for adler32 (adler32-zlib) [ 3332.753103] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3332.850601] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3334.431157] Key type lgssc registered [ 3334.805914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3337.489851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3337.500809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.663767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3340.149435] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3343.924299] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.987983] LustreError: 155738:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3345.993600] LustreError: 155738:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3345.997285] LustreError: 155738:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3346.125757] Lustre: server umount lustre-MDT0000 complete [ 3351.186262] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3354.631148] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3366.535650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.618651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3366.628919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3366.665941] Lustre: lustre-MDT0000: new disk, initializing [ 3366.688225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.694939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3368.040437] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3372.260194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.300541] Lustre: 159031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3372.328570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3372.331133] Lustre: Skipped 1 previous similar message [ 3372.369872] Lustre: lustre-MDT0001: new disk, initializing [ 3372.406273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.420106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3372.424468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3373.784635] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3377.509508] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3379.510671] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3381.904510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3382.024320] Lustre: lustre-OST0000: new disk, initializing [ 3382.026297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3382.030676] Lustre: 160999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3382.069536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.146094] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3386.874644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3386.878146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3386.920337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3387.949441] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3389.786536] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3390.337220] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:28:28 (1783074508) [ 3468.767703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.767881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.772408] Lustre: Skipped 2 previous similar messages [ 3468.775138] Lustre: Skipped 1 previous similar message [ 3472.498323] Lustre: server umount lustre-MDT0000 complete [ 3473.745586] LustreError: 159024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074592 with bad export cookie 1348840200448400703 [ 3473.746491] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3473.749422] LustreError: 159024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3473.887402] LustreError: 159039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.887432] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.892619] LustreError: 159039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3473.896337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3479.007813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3479.007810] LustreError: 159038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.103537] LustreError: 159039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.103544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3488.223126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3488.223367] LustreError: 159038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3488.223370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3488.295771] Lustre: server umount lustre-MDT0001 complete [ 3495.755702] Lustre: server umount lustre-OST0000 complete [ 3498.726349] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3502.107297] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3514.010022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.086374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3514.096336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3514.135526] Lustre: lustre-MDT0000: new disk, initializing [ 3514.155869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.162261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3515.447266] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3519.614138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.653047] Lustre: 165641:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3519.676729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3519.678394] Lustre: Skipped 1 previous similar message [ 3519.714615] Lustre: lustre-MDT0001: new disk, initializing [ 3519.756595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3519.759637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3521.149600] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3524.819428] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3526.559641] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3528.833728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3528.929851] Lustre: lustre-OST0000: new disk, initializing [ 3528.932430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3528.935664] Lustre: 167606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3528.967059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.969058] Lustre: Skipped 1 previous similar message [ 3530.885188] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3532.536179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3532.540185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3532.573292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3534.601531] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3536.063504] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3536.132528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3537.425404] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3537.497770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3543.009928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3543.010351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.014440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.017638] Lustre: Skipped 1 previous similar message [ 3544.210862] Lustre: server umount lustre-OST0000 complete [ 3548.128790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.133509] Lustre: Skipped 1 previous similar message [ 3551.762545] Lustre: server umount lustre-MDT0000 complete [ 3552.955088] LustreError: 167605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074672 with bad export cookie 1348840200448559519 [ 3552.957316] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3552.958746] LustreError: 167605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3553.069089] Lustre: server umount lustre-MDT0001 complete [ 3555.768544] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:31:14 (1783074674) [ 3561.030877] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3564.170791] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3576.932968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.025312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3577.035626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3577.079113] Lustre: lustre-MDT0000: new disk, initializing [ 3577.118194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.133832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3578.643495] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3583.342974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.389959] Lustre: 172355:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3583.506462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3585.057890] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3589.525061] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3591.672713] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3594.325424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.434605] Lustre: 174326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3596.733126] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3600.721905] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3600.893315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3600.938433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3611.106237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3611.106667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.109156] LustreError: Skipped 1 previous similar message [ 3611.114479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.118022] Lustre: Skipped 4 previous similar messages [ 3614.224956] Lustre: server umount lustre-OST0000 complete [ 3621.599312] Lustre: server umount lustre-MDT0000 complete [ 3622.863941] LustreError: 173153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074741 with bad export cookie 1348840200448560401 [ 3622.865534] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3622.868523] LustreError: 173153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3622.983550] Lustre: server umount lustre-MDT0001 complete [ 3625.758950] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:32:23 (1783074743) [ 3631.210969] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3634.709580] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3647.222100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.343805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3647.358835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3647.361378] Lustre: Skipped 3 previous similar messages [ 3647.400121] Lustre: lustre-MDT0000: new disk, initializing [ 3647.402448] Lustre: Skipped 2 previous similar messages [ 3647.432498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.434665] Lustre: Skipped 2 previous similar messages [ 3647.440962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3647.445257] Lustre: Skipped 2 previous similar messages [ 3648.821870] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3653.172241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.222865] Lustre: 178642:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3653.351284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3654.770127] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3658.799142] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3660.675075] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3663.139475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.257988] Lustre: 180618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3665.238730] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3668.922242] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3671.584200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3676.640390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.641146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.642703] LustreError: Skipped 2 previous similar messages [ 3676.646471] Lustre: Skipped 3 previous similar messages [ 3676.647133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.651499] Lustre: Skipped 5 previous similar messages [ 3678.153553] Lustre: server umount lustre-OST0000 complete [ 3681.425320] Lustre: server umount lustre-MDT0000 complete [ 3681.761285] LustreError: 178653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.767813] LustreError: 178653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3682.678069] LustreError: 179442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074801 with bad export cookie 1348840200448561472 [ 3682.681439] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.683276] LustreError: 179442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3682.794639] Lustre: server umount lustre-MDT0001 complete [ 3686.453456] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3687.452843] Key type lgssc unregistered [ 3687.579387] LNet: 182469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3687.582361] LNetError: 182469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3687.593264] LNet: Removed LNI 192.168.201.152@tcp [ 3687.913427] Key type .llcrypt unregistered [ 3687.914418] Key type ._llcrypt unregistered [ 3696.928260] Key type ._llcrypt registered [ 3696.929523] Key type .llcrypt registered [ 3696.970735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3697.414568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3697.470877] alg: No test for adler32 (adler32-zlib) [ 3698.330745] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3698.418422] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3699.999136] Key type lgssc registered [ 3700.355990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3702.832430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3702.839080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.974151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3705.301619] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3708.201784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.324215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.661932] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3712.193224] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3714.020957] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3716.402747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.514891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.467501] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3720.861396] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3729.833354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3744.224965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3744.227863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.232577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3745.250547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3745.252513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.253243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3749.137200] Lustre: server umount lustre-OST0000 complete [ 3754.463892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3754.466589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.471152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3754.473072] Lustre: Skipped 1 previous similar message [ 3756.622614] Lustre: server umount lustre-MDT0000 complete [ 3757.900088] LustreError: 183843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074876 with bad export cookie 10607854884710830551 [ 3757.903127] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3757.904404] LustreError: 183843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3758.015663] Lustre: server umount lustre-MDT0001 complete [ 3761.935530] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3763.044435] Key type lgssc unregistered [ 3763.179407] LNet: 187413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3763.181994] LNetError: 187413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3763.190342] LNet: Removed LNI 192.168.201.152@tcp [ 3763.516940] Key type .llcrypt unregistered [ 3763.518202] Key type ._llcrypt unregistered [ 3773.166272] Key type ._llcrypt registered [ 3773.167270] Key type .llcrypt registered [ 3773.202648] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3773.604456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3773.613341] alg: No test for adler32 (adler32-zlib) [ 3774.464616] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 3774.553331] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3776.135170] Key type lgssc registered [ 3776.516160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3779.172339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3779.181245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.343927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.729874] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3784.749568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.885340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3786.295653] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3788.852050] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3790.782253] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3793.123143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.246963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.216594] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3797.749353] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3805.541584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3810.784390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.787245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.789211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3810.792885] Lustre: Skipped 1 previous similar message [ 3813.252306] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 3813.254165] Lustre: Skipped 1 previous similar message [ 3815.906814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.910291] Lustre: Skipped 1 previous similar message [ 3816.527323] Lustre: server umount lustre-OST0000 complete [ 3818.937967] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91c382c81000.ost_server_uuid 50 [ 3819.493765] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3831.264796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3831.264807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.265492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.273461] Lustre: Skipped 1 previous similar message [ 3835.664864] Lustre: server umount lustre-MDT0000 complete [ 3836.385024] LustreError: 188807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.390738] LustreError: 188807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3836.934175] LustreError: 188832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783074955 with bad export cookie 9476652553561483357 [ 3836.936939] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.937952] LustreError: 188832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3837.054115] Lustre: server umount lustre-MDT0001 complete [ 3839.869908] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:35:58 (1783074958) [ 3842.037307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.184844] LustreError: 192717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.208064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.472978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3846.279463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.400810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.719259] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3850.272258] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3852.088347] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3854.340491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3854.482135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.499613] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3858.964159] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3861.381056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3861.463654] Lustre: lustre-OST0001: new disk, initializing [ 3861.465703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3861.469136] Lustre: 195554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3863.042355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3863.046173] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3863.091078] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3863.590605] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3867.174701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3867.470546] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3872.227926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3872.229092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.231264] LustreError: Skipped 1 previous similar message [ 3872.236314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3872.238888] Lustre: Skipped 2 previous similar messages [ 3874.491195] Lustre: server umount lustre-OST0000 complete [ 3876.498943] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3876.582418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3877.348309] LustreError: 194537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.352653] LustreError: 194537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3878.021815] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3878.095555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3882.468342] LustreError: 195809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.473667] LustreError: 195809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3887.423808] LustreError: 194537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.431032] LustreError: 194537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3888.608190] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.608317] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.611291] LustreError: Skipped 1 previous similar message [ 3888.615943] Lustre: Skipped 1 previous similar message [ 3888.618582] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3888.622205] Lustre: Skipped 2 previous similar messages [ 3894.221147] Lustre: server umount lustre-OST0001 complete [ 3897.825190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.825250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3901.591613] Lustre: server umount lustre-MDT0000 complete [ 3902.787107] LustreError: 192700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075021 with bad export cookie 9476652553561484414 [ 3902.789390] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.790253] LustreError: 192700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3902.895360] Lustre: server umount lustre-MDT0001 complete [ 3910.608843] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:37:08 (1783075028) [ 3912.948080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.068661] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3913.084411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3913.174322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.177015] Lustre: Skipped 1 previous similar message [ 3914.650353] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3917.491926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.527566] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3917.538450] Lustre: 198968:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3918.936799] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3921.353934] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.199757] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3925.505901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3925.576677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3927.244732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3927.586471] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3929.944445] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3932.155427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.178469] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3934.089894] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3936.378697] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3937.763670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.765232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.765910] LustreError: Skipped 1 previous similar message [ 3937.769364] Lustre: Skipped 1 previous similar message [ 3937.771588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.774168] Lustre: Skipped 5 previous similar messages [ 3943.359440] Lustre: server umount lustre-OST0000 complete [ 3946.877226] LustreError: 200817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3946.882859] LustreError: 200817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3958.241093] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3958.242927] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.245924] Lustre: Skipped 1 previous similar message [ 3960.076542] Lustre: server umount lustre-OST0001 complete [ 3967.448361] Lustre: server umount lustre-MDT0000 complete [ 3968.480033] LustreError: 199718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.485666] LustreError: 199718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3968.695365] LustreError: 199712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075087 with bad export cookie 9476652553561485800 [ 3968.698812] LustreError: 199712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3968.698878] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.802370] Lustre: server umount lustre-MDT0001 complete [ 3976.326528] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:38:14 (1783075094) [ 3981.284448] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 3984.408178] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3995.398957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3995.474608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3995.483118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3995.519624] Lustre: lustre-MDT0000: new disk, initializing [ 3995.539463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.541567] Lustre: Skipped 3 previous similar messages [ 3995.546058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3996.833930] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4000.771739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.810574] Lustre: 207025:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4000.831195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4000.832837] Lustre: Skipped 1 previous similar message [ 4000.865967] Lustre: lustre-MDT0001: new disk, initializing [ 4000.900200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4000.903777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4002.173564] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4005.723527] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4007.424358] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4009.605853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4009.709928] Lustre: lustre-OST0000: new disk, initializing [ 4009.711613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4009.714983] Lustre: 208990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4011.678169] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4011.707118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4011.711073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4011.748908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4015.394896] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4016.819890] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4016.895937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4018.316572] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4018.398354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4022.245505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4022.246328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.247812] LustreError: Skipped 2 previous similar messages [ 4022.251584] Lustre: Skipped 2 previous similar messages [ 4022.252590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4022.256753] Lustre: Skipped 8 previous similar messages [ 4025.165977] Lustre: server umount lustre-OST0000 complete [ 4032.537747] Lustre: server umount lustre-MDT0000 complete [ 4033.782213] LustreError: 208989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075152 with bad export cookie 9476652553561487347 [ 4033.784904] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.785867] LustreError: 208989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4033.902678] Lustre: server umount lustre-MDT0001 complete [ 4036.783696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.935856] LustreError: 211206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.365421] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4041.312579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.859735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4045.376962] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4047.351885] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4049.762840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.861970] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4054.406420] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4055.898877] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4063.119526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4064.554681] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4064.630067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4071.383224] Lustre: server umount lustre-OST0000 complete [ 4073.466677] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4074.801206] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4098.425727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4098.532363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.535406] Lustre: Skipped 5 previous similar messages [ 4100.435745] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4102.774306] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4103.343505] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4106.377651] Lustre: server umount lustre-OST0000 complete [ 4108.255743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4108.258415] LustreError: Skipped 3 previous similar messages [ 4108.259934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4108.264373] Lustre: Skipped 5 previous similar messages [ 4108.266409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4108.268528] Lustre: Skipped 7 previous similar messages [ 4109.655364] Lustre: server umount lustre-MDT0000 complete [ 4110.958743] LustreError: 211231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075230 with bad export cookie 9476652553561488215 [ 4110.964222] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.090939] Lustre: server umount lustre-MDT0001 complete [ 4113.994285] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:40:32 (1783075232) [ 4116.113879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.266125] LustreError: 217210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.653229] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4120.484068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.990369] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4124.484961] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4126.251911] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4128.432398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.371965] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4132.703278] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4134.115379] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4142.345470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4143.673088] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4143.740853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4145.941179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.013957] Lustre: lustre-OST0001: new disk, initializing [ 4146.015747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4146.018070] Lustre: 220632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4147.816326] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4151.295087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4151.298213] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4151.330126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4151.422376] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4152.711558] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4152.786772] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4154.020446] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4154.088332] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4160.766856] Lustre: server umount lustre-OST0001 complete [ 4162.637782] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4162.705238] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4164.138032] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4164.211254] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4187.903868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4189.838204] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4192.234333] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4192.798897] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4193.253738] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4193.257914] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4193.258064] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4193.264661] Lustre: Skipped 1 previous similar message [ 4199.546892] Lustre: server umount lustre-OST0001 complete [ 4217.490212] Lustre: server umount lustre-OST0000 complete [ 4224.855780] Lustre: server umount lustre-MDT0000 complete [ 4226.005214] LustreError: 220630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075345 with bad export cookie 9476652553561489132 [ 4226.008020] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.008636] LustreError: 220630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4240.351095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4240.453937] Lustre: server umount lustre-MDT0001 complete [ 4248.287616] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:42:46 (1783075366) [ 4250.382650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.478409] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4250.491159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4250.559031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.561259] Lustre: Skipped 5 previous similar messages [ 4251.907191] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4255.103608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.143471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4255.154274] Lustre: 225866:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4256.726031] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4259.293165] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4261.029863] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4263.231097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.297567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4265.276843] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4267.639302] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4271.500726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4271.526308] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4273.431170] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4275.733238] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4276.257808] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4277.593764] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4277.664109] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4278.904834] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4278.975359] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4280.646053] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91c38b2ff800.ost_server_uuid 50 [ 4281.128710] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91c38b2ff800.ost_server_uuid in FULL state after 0 sec [ 4281.431506] Lustre: Permanently deactivating lustre-OST0001 [ 4281.436520] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4282.158939] Lustre: Permanently reactivating lustre-OST0001 [ 4284.899881] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4284.900746] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.902282] LustreError: Skipped 7 previous similar messages [ 4284.905722] Lustre: Skipped 8 previous similar messages [ 4284.907867] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4284.910259] Lustre: Skipped 13 previous similar messages [ 4289.151875] Lustre: server umount lustre-OST0001 complete [ 4290.020559] LustreError: 227710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.024738] LustreError: 227710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 4296.525219] Lustre: server umount lustre-OST0000 complete [ 4303.827372] Lustre: server umount lustre-MDT0000 complete [ 4304.991042] LustreError: 228744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075424 with bad export cookie 9476652553561490406 [ 4304.994222] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.994701] LustreError: 228744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4305.093305] Lustre: server umount lustre-MDT0001 complete [ 4312.275102] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:43:50 (1783075430) [ 4314.568343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.658676] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4314.668415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4314.671088] Lustre: Skipped 5 previous similar messages [ 4315.969355] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4318.798771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4318.837252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4318.846489] Lustre: 232545:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4320.250989] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4322.656627] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.504468] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4326.762029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4326.831849] Lustre: Permanently deactivating lustre-OST0000 [ 4326.836784] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4328.708088] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4330.973129] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4333.037292] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4333.093120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4334.935662] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4337.223277] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4339.219830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4343.085606] Lustre: Permanently reactivating lustre-OST0000 [ 4343.091255] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4343.093898] Lustre: Skipped 2 previous similar messages [ 4345.458891] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4345.480098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4355.200756] Lustre: server umount lustre-OST0001 complete [ 4362.573775] Lustre: server umount lustre-OST0000 complete [ 4378.079154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4378.139092] Lustre: server umount lustre-MDT0000 complete [ 4379.379132] LustreError: 236142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075498 with bad export cookie 9476652553561492065 [ 4379.382507] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.382775] LustreError: 236142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4379.511176] Lustre: server umount lustre-MDT0001 complete [ 4383.247942] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4384.223407] Key type lgssc unregistered [ 4384.348329] LNet: 237404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4384.350378] LNetError: 237404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4384.356245] LNet: Removed LNI 192.168.201.152@tcp [ 4384.652109] Key type .llcrypt unregistered [ 4384.653542] Key type ._llcrypt unregistered [ 4392.948554] Key type ._llcrypt registered [ 4392.949954] Key type .llcrypt registered [ 4392.985957] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4393.324856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.346915] alg: No test for adler32 (adler32-zlib) [ 4394.182038] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4394.266473] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4395.839171] Key type lgssc registered [ 4396.176661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4399.418621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4399.424042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.563591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.830716] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4404.562157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.598825] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4404.608990] Lustre: 238854:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4404.613656] Lustre: Permanently deactivating lustre-MDT0001 [ 4404.618169] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4404.719705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.979314] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4408.226518] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4409.842984] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4409.950371] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4412.121586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4412.228413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.238448] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4414.304719] LustreError: 241078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.310396] LustreError: 241078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4414.311851] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4416.467293] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4418.651612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4418.718215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4420.537666] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4422.725464] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4424.164911] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4425.191354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4425.191828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4428.568148] Lustre: Permanently reactivating lustre-MDT0001 [ 4428.572552] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4428.574340] Lustre: Skipped 2 previous similar messages [ 4432.768948] Lustre: lustre-MDT0001: Client 86f79292-5f40-4216-9248-3b5d2c1b2d2f (at 192.168.201.52@tcp) reconnecting [ 4434.433312] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.438606] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 4434.442922] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4434.447049] LustreError: 242517:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 4434.447458] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4434.449951] LustreError: 242517:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 4434.454101] LustreError: 242517:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.152@tcp: cfg command failed: rc = -108 [ 4434.456561] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 4434.456561] [ 4434.459054] LustreError: 238857:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4435.335446] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4435.826944] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4437.375078] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4437.438404] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4438.695828] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4438.767402] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4439.575495] Lustre: Permanently deactivating lustre-MDT0001 [ 4439.581919] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4439.585032] Lustre: Skipped 2 previous similar messages [ 4441.338895] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4445.665515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4445.665813] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.666919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.673869] Lustre: Skipped 3 previous similar messages [ 4449.052223] Lustre: server umount lustre-MDT0000 complete [ 4450.287437] LustreError: 238846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075569 with bad export cookie 17204637946512515766 [ 4450.293169] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.423843] Lustre: server umount lustre-MDT0001 complete [ 4457.788303] Lustre: server umount lustre-OST0000 complete [ 4461.515389] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:46:19 (1783075579) [ 4464.821193] Lustre: server umount lustre-OST0001 complete [ 4466.664806] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4469.572978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4480.303230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.378722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4480.382268] Lustre: Skipped 2 previous similar messages [ 4480.389665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4480.421794] Lustre: lustre-MDT0000: new disk, initializing [ 4480.439512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.444470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4481.655290] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4485.473926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.507598] Lustre: 247109:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4485.526556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4485.528744] Lustre: Skipped 1 previous similar message [ 4485.561335] Lustre: lustre-MDT0001: new disk, initializing [ 4485.600682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4485.604535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4486.748212] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4490.317047] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4492.047309] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4494.205822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4494.299419] Lustre: lustre-OST0000: new disk, initializing [ 4494.301296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4494.304735] Lustre: 249075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4494.330620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.333439] Lustre: Skipped 1 previous similar message [ 4495.798160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4495.801582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4495.830953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4496.130588] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4499.628567] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4500.235075] LustreError: 247113:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4504.436489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.498867] Lustre: lustre-OST0001: new disk, initializing [ 4504.501072] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4504.503372] Lustre: 250188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4506.370779] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4509.824710] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4510.322637] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4510.327056] LustreError: 247113:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4510.336477] LustreError: 250268:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4513.277527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4513.280355] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4513.307923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4520.431072] LustreError: 250268:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4522.372260] Lustre: lustre-OST0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 4522.375169] Lustre: Skipped 3 previous similar messages [ 4523.492354] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4523.492375] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.495766] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4523.499174] Lustre: Skipped 1 previous similar message [ 4527.356150] Lustre: server umount lustre-OST0001 complete [ 4527.484638] LustreError: 249059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.610806] LustreError: 249367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.615068] LustreError: 249367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4533.733137] LustreError: 249061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.741087] LustreError: 249061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4538.852216] LustreError: 249367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.856319] LustreError: 249367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4541.920330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4541.920483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.922680] LustreError: Skipped 2 previous similar messages [ 4541.927077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.929731] Lustre: Skipped 2 previous similar messages [ 4545.355288] Lustre: server umount lustre-OST0000 complete [ 4547.039880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4547.041904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.045659] Lustre: Skipped 1 previous similar message [ 4547.047679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4547.049126] Lustre: Skipped 2 previous similar messages [ 4552.737827] Lustre: server umount lustre-MDT0000 complete [ 4554.022042] LustreError: 247100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075673 with bad export cookie 17204637946512518566 [ 4554.024966] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4554.025683] LustreError: 247100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4560.173878] Lustre: server umount lustre-MDT0001 complete [ 4563.628292] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4564.589915] Key type lgssc unregistered [ 4564.717467] LNet: 252241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4564.719868] LNetError: 252241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4564.727480] LNet: Removed LNI 192.168.201.152@tcp [ 4565.015834] Key type .llcrypt unregistered [ 4565.016864] Key type ._llcrypt unregistered [ 4578.401487] Key type ._llcrypt registered [ 4578.402756] Key type .llcrypt registered [ 4578.435606] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4578.691736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4578.701732] alg: No test for adler32 (adler32-zlib) [ 4579.544752] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4579.621270] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4581.191157] Key type lgssc registered [ 4581.519296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4583.827497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4583.832815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.905583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4584.911157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4584.960823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.186415] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4588.819918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.851075] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4588.862740] Lustre: 254424:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4588.959437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4590.306287] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4592.540414] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4594.238789] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4596.360985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4596.424562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4596.484026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.847545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4598.381770] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4600.665777] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4643.809935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.810318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.813074] LustreError: Skipped 1 previous similar message [ 4643.819478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4643.821705] Lustre: Skipped 1 previous similar message [ 4645.912925] Lustre: server umount lustre-OST0000 complete [ 4647.466476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4651.861819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4651.969597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.678166] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4655.932407] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4663.270122] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.275920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.275950] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4663.284465] Lustre: Skipped 1 previous similar message [ 4693.987408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.987426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4693.989609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4693.991163] Lustre: Skipped 2 previous similar messages [ 4693.993488] LustreError: Skipped 1 previous similar message [ 4695.598879] Lustre: server umount lustre-OST0000 complete [ 4699.104936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.105504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4699.118521] Lustre: Skipped 1 previous similar message [ 4699.120805] Lustre: Skipped 1 previous similar message [ 4704.224126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.226896] Lustre: Skipped 1 previous similar message [ 4709.343939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.346681] Lustre: Skipped 1 previous similar message [ 4711.391096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4711.450850] Lustre: server umount lustre-MDT0000 complete [ 4712.652227] LustreError: 256281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075831 with bad export cookie 10369522556547560681 [ 4712.655582] LustreError: 256281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4712.655613] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4712.770525] Lustre: server umount lustre-MDT0001 complete [ 4716.228235] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4717.303456] Key type lgssc unregistered [ 4717.426410] LNet: 259833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4717.429260] LNetError: 259833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4717.442467] LNet: Removed LNI 192.168.201.152@tcp [ 4717.728746] Key type .llcrypt unregistered [ 4717.730027] Key type ._llcrypt unregistered [ 4726.669950] Key type ._llcrypt registered [ 4726.670946] Key type .llcrypt registered [ 4726.707139] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4727.167448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4727.219517] alg: No test for adler32 (adler32-zlib) [ 4728.068123] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4728.159749] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4729.751158] Key type lgssc registered [ 4730.130746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4732.596720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4732.602903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.739578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.033282] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4737.765146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.885544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4739.243708] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4741.645675] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4743.422459] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4745.603445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4745.714483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.669034] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4750.070532] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.989604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4768.224396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4768.226750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.228101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4768.232065] Lustre: Skipped 1 previous similar message [ 4773.345658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4773.347186] Lustre: Skipped 2 previous similar messages [ 4773.525603] Lustre: server umount lustre-OST0000 complete [ 4778.464583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.465178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.469855] Lustre: Skipped 1 previous similar message [ 4783.584388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.586573] Lustre: Skipped 2 previous similar messages [ 4788.704447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.706342] Lustre: Skipped 1 previous similar message [ 4789.215090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4789.270570] Lustre: server umount lustre-MDT0000 complete [ 4790.452227] LustreError: 261207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075909 with bad export cookie 5246435646973354093 [ 4790.454809] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4790.456191] LustreError: 261207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4790.562209] Lustre: server umount lustre-MDT0001 complete [ 4794.102903] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4795.044944] Key type lgssc unregistered [ 4795.179330] LNet: 265551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4795.181959] LNetError: 265551:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4796.197294] LNet: Removed LNI 192.168.201.152@tcp [ 4796.468838] Key type .llcrypt unregistered [ 4796.469821] Key type ._llcrypt unregistered [ 4803.480056] Key type ._llcrypt registered [ 4803.481113] Key type .llcrypt registered [ 4803.516329] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4803.870254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4803.880382] alg: No test for adler32 (adler32-zlib) [ 4804.732213] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4804.829318] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4806.407152] Key type lgssc registered [ 4806.770731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4809.391143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4809.396859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.556984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.998136] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4814.798319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.933161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4816.383816] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4818.780676] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4820.744495] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4823.002120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4823.140080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.150968] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4827.624806] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4836.455565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4841.953368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4841.957465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.963161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4841.965256] Lustre: Skipped 1 previous similar message [ 4847.073172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4847.075593] Lustre: Skipped 1 previous similar message [ 4852.194617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4852.196895] Lustre: Skipped 1 previous similar message [ 4856.287089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4856.341189] Lustre: server umount lustre-OST0000 complete [ 4860.896141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4860.898492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.901748] Lustre: Skipped 1 previous similar message [ 4860.904031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4862.433087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.772868] Lustre: server umount lustre-MDT0000 complete [ 4865.136262] LustreError: 266729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783075984 with bad export cookie 17918562529177469877 [ 4865.139152] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4865.139798] LustreError: 266729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4865.284334] Lustre: server umount lustre-MDT0001 complete [ 4868.880299] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4869.843154] Key type lgssc unregistered [ 4869.969366] LNet: 270445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4869.972304] LNetError: 270445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4869.983284] LNet: Removed LNI 192.168.201.152@tcp [ 4870.274033] Key type .llcrypt unregistered [ 4870.274970] Key type ._llcrypt unregistered [ 4879.149257] Key type ._llcrypt registered [ 4879.150588] Key type .llcrypt registered [ 4879.194159] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4879.677159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4879.715845] alg: No test for adler32 (adler32-zlib) [ 4880.586876] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4880.676252] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4882.255120] Key type lgssc registered [ 4882.643261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4885.487787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4885.494615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.669179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.936012] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4890.708910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.814504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4892.070346] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4894.372333] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4896.276502] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4898.756821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4898.890088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.769574] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4903.148552] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.206226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4921.826096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4921.826209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.828594] LustreError: Skipped 1 previous similar message [ 4921.833218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.836903] Lustre: Skipped 1 previous similar message [ 4923.147683] Lustre: server umount lustre-OST0000 complete [ 4926.944422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4926.944931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.945421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.954472] Lustre: Skipped 2 previous similar messages [ 4930.529990] Lustre: server umount lustre-MDT0000 complete [ 4931.760217] LustreError: 273676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076050 with bad export cookie 4624976921157373722 [ 4931.763212] LustreError: 273676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4931.764090] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4931.885532] Lustre: server umount lustre-MDT0001 complete [ 4935.389279] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4936.508829] Key type lgssc unregistered [ 4936.649476] LNet: 276157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4936.653826] LNetError: 276157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4936.661430] LNet: Removed LNI 192.168.201.152@tcp [ 4936.962843] Key type .llcrypt unregistered [ 4936.964243] Key type ._llcrypt unregistered [ 4944.155321] Key type ._llcrypt registered [ 4944.156611] Key type .llcrypt registered [ 4944.202073] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4944.578778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4944.608356] alg: No test for adler32 (adler32-zlib) [ 4945.491913] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 4945.604418] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4947.191217] Key type lgssc registered [ 4947.648568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4950.297467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4950.303205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.462031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4952.758338] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4955.727888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.873264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.277132] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4959.711334] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4961.562584] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4963.792043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.916733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4965.798750] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4968.161915] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4976.229883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4985.823990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4985.828268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4985.835488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4986.851415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4986.851850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.858873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4988.110989] Lustre: server umount lustre-OST0000 complete [ 4991.970054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4991.970064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.976758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4991.978826] Lustre: Skipped 2 previous similar messages [ 4995.489858] Lustre: server umount lustre-MDT0000 complete [ 4996.722489] LustreError: 277337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076115 with bad export cookie 10397252499091858249 [ 4996.726857] LustreError: 277337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4996.727427] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4996.849428] Lustre: server umount lustre-MDT0001 complete [ 5000.626394] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5001.686768] Key type lgssc unregistered [ 5001.820519] LNet: 281088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5001.823266] LNetError: 281088:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5001.832550] LNet: Removed LNI 192.168.201.152@tcp [ 5002.188363] Key type .llcrypt unregistered [ 5002.189441] Key type ._llcrypt unregistered [ 5006.075765] Key type ._llcrypt registered [ 5006.077498] Key type .llcrypt registered [ 5006.118513] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5011.744772] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5017.906826] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5018.379972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5018.396898] alg: No test for adler32 (adler32-zlib) [ 5019.256856] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5019.363168] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5020.951165] Key type lgssc registered [ 5021.336175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5032.626385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5032.632844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.715532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5033.726796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5033.763410] Lustre: lustre-MDT0000: new disk, initializing [ 5033.785983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.792804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5035.168208] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5039.344495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.393866] Lustre: 284612:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5039.426886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5039.429797] Lustre: Skipped 1 previous similar message [ 5039.468711] Lustre: lustre-MDT0001: new disk, initializing [ 5039.527166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5039.541768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5039.547289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5040.983790] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5044.838304] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5046.692227] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5048.982116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.090661] Lustre: lustre-OST0000: new disk, initializing [ 5049.092417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5049.094718] Lustre: 286577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5049.120711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.110110] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5054.455209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5054.458907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5054.483449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5054.635089] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5055.942990] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5056.013264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5057.304877] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5057.385919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5059.554785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5059.554816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.555492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5059.563014] Lustre: Skipped 1 previous similar message [ 5064.082544] Lustre: server umount lustre-OST0000 complete [ 5069.792474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.793238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5069.797556] Lustre: Skipped 1 previous similar message [ 5069.799327] Lustre: Skipped 1 previous similar message [ 5071.451725] Lustre: server umount lustre-MDT0000 complete [ 5072.611980] LustreError: 284604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076191 with bad export cookie 15512439637992329982 [ 5072.614950] LustreError: 284604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5072.615566] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.725262] Lustre: server umount lustre-MDT0001 complete [ 5075.332358] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:56:33 (1783076193) [ 5077.454387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.633184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.918820] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5081.684568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.814561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5083.120924] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5085.317844] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5086.960876] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5088.973269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5090.787570] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5092.971170] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5106.655687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5106.657818] LustreError: Skipped 1 previous similar message [ 5106.659082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.663028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.665310] Lustre: Skipped 1 previous similar message [ 5107.680942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.268706] Lustre: server umount lustre-OST0000 complete [ 5116.896755] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5116.898857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.901965] Lustre: Skipped 1 previous similar message [ 5116.903568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.905972] Lustre: Skipped 2 previous similar messages [ 5119.761759] Lustre: server umount lustre-MDT0000 complete [ 5120.958571] LustreError: 288967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076240 with bad export cookie 15512439637992330864 [ 5120.961102] LustreError: 289723:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5120.963046] LustreError: 288967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5120.964755] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.092258] Lustre: server umount lustre-MDT0001 complete [ 5124.507436] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5125.488796] Key type lgssc unregistered [ 5125.609331] LNet: 292539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5125.611290] LNetError: 292539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5125.617356] LNet: Removed LNI 192.168.201.152@tcp [ 5125.893676] Key type .llcrypt unregistered [ 5125.894616] Key type ._llcrypt unregistered [ 5134.289350] Key type ._llcrypt registered [ 5134.291081] Key type .llcrypt registered [ 5134.325517] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5140.237900] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5140.621898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5140.635529] alg: No test for adler32 (adler32-zlib) [ 5141.480317] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5141.566095] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5143.143141] Key type lgssc registered [ 5143.461969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5156.867974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5156.873771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.942975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5157.951648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5157.983260] Lustre: lustre-MDT0000: new disk, initializing [ 5158.000667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.006588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5159.174676] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5163.093850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.137315] Lustre: 296336:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5163.160292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5163.162019] Lustre: Skipped 1 previous similar message [ 5163.193714] Lustre: lustre-MDT0001: new disk, initializing [ 5163.226311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5163.237527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5163.241379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5164.463306] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5168.012290] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5169.683164] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5171.784445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.876801] Lustre: lustre-OST2710: new disk, initializing [ 5171.878530] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5171.881743] Lustre: 298301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5171.908754] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5173.669914] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5173.812458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5173.816496] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5173.848427] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5177.245919] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5179.396664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.453588] Lustre: lustre-OST03e8: new disk, initializing [ 5179.455351] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5179.458391] Lustre: 299362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5179.485619] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5181.280513] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5181.636528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5181.639801] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5181.667371] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5183.546617] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5186.290532] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5186.357785] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5187.765461] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5187.845637] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5189.210122] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5189.268102] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5190.485487] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5190.555867] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5192.161781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.164480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.165648] Lustre: Skipped 3 previous similar messages [ 5192.169315] Lustre: Skipped 3 previous similar messages [ 5197.280172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.282899] Lustre: Skipped 3 previous similar messages [ 5197.721757] Lustre: server umount lustre-MDT0000 complete [ 5198.924709] LustreError: 296328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076317 with bad export cookie 6140213892281481580 [ 5198.928146] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.929169] LustreError: 296328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5199.052658] Lustre: server umount lustre-MDT0001 complete [ 5210.633805] Lustre: server umount lustre-OST2710 complete [ 5222.230500] Lustre: server umount lustre-OST03e8 complete [ 5227.737797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5230.850096] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5241.946569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.022971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5242.033760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5242.077466] Lustre: lustre-MDT0000: new disk, initializing [ 5242.098986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.105213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5243.405850] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5247.418633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.466915] Lustre: 304598:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5247.581778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5248.871350] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5252.428114] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5254.231162] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5256.450497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5256.547818] Lustre: lustre-OST0000: new disk, initializing [ 5256.549323] Lustre: Skipped 1 previous similar message [ 5256.550915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5256.552666] Lustre: Skipped 2 previous similar messages [ 5256.554704] Lustre: 306566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5256.581732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5256.583830] Lustre: Skipped 1 previous similar message [ 5258.471876] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5262.022494] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5262.838731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5262.841732] Lustre: Skipped 1 previous similar message [ 5262.843757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5262.869316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5263.384852] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5263.449183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5264.655451] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5264.714062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5267.938340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.939218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5267.943753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.946014] LustreError: Skipped 1 previous similar message [ 5267.949988] Lustre: Skipped 1 previous similar message [ 5271.438466] Lustre: server umount lustre-OST0000 complete [ 5273.056669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.056911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5273.062221] Lustre: Skipped 1 previous similar message [ 5273.062815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5273.067969] Lustre: Skipped 1 previous similar message [ 5278.175878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5278.178174] Lustre: Skipped 1 previous similar message [ 5278.739976] Lustre: server umount lustre-MDT0000 complete [ 5279.916984] LustreError: 304592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076398 with bad export cookie 6140213892281483246 [ 5279.920437] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5279.921908] LustreError: 304592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5280.030903] Lustre: server umount lustre-MDT0001 complete [ 5282.706139] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:00:00 (1783076400) [ 5283.204054] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5283.794202] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:00:02 (1783076402) [ 5284.677733] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5293.060175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.159661] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5293.168338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5293.236178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.674779] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5297.530163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.558520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5297.567479] Lustre: 310417:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5299.100732] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5314.849963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5314.910777] Lustre: Denying initial registration attempt from nid 192.168.201.152@tcp, specified as failover [ 5314.913893] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5314.916699] LustreError: 311869:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5314.918765] LustreError: 311869:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5314.920585] LustreError: 311869:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5314.941442] Lustre: server umount lustre-OST0000 complete [ 5314.942866] LustreError: 311869:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5318.111755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5318.114871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.119432] Lustre: Skipped 1 previous similar message [ 5318.121386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.087815] Lustre: server umount lustre-MDT0000 complete [ 5324.244210] LustreError: 310452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076443 with bad export cookie 6140213892281484114 [ 5324.246860] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5324.248763] LustreError: 310452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5324.335148] Lustre: server umount lustre-MDT0001 complete [ 5327.816555] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5328.730664] Key type lgssc unregistered [ 5328.852411] LNet: 312987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5328.854894] LNetError: 312987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5328.865265] LNet: Removed LNI 192.168.201.152@tcp [ 5329.144281] Key type .llcrypt unregistered [ 5329.145509] Key type ._llcrypt unregistered [ 5332.465863] Key type ._llcrypt registered [ 5332.466814] Key type .llcrypt registered [ 5332.500074] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5332.750905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5332.757577] alg: No test for adler32 (adler32-zlib) [ 5333.594990] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5333.672583] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5335.247126] Key type lgssc registered [ 5335.608305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5359.541138] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5362.457349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5362.464988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.550611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5363.557418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5363.619355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.022497] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5367.863078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.895753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5367.910326] Lustre: 315739:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5368.012635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5369.353248] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5371.771039] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5373.585620] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5388.897987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5388.967456] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5389.032240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.882028] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5393.241654] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5393.990651] Lustre: server umount lustre-OST0000 complete [ 5399.007824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5399.010841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.015461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.520287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.526715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5401.368755] Lustre: server umount lustre-MDT0000 complete [ 5402.593027] LustreError: 315730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076521 with bad export cookie 9763236063536490718 [ 5402.596255] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5402.596976] LustreError: 315730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5402.706521] Lustre: server umount lustre-MDT0001 complete [ 5406.437689] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5407.525723] Key type lgssc unregistered [ 5407.646398] LNet: 319304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5407.649112] LNetError: 319304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5407.658278] LNet: Removed LNI 192.168.201.152@tcp [ 5407.968583] Key type .llcrypt unregistered [ 5407.969762] Key type ._llcrypt unregistered [ 5417.377204] Key type ._llcrypt registered [ 5417.378313] Key type .llcrypt registered [ 5417.414117] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5417.769659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5417.811093] alg: No test for adler32 (adler32-zlib) [ 5418.681749] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5418.772557] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5420.351168] Key type lgssc registered [ 5420.737118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5423.335698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5423.342028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.480320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.873046] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5428.725452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.848891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.300854] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5432.786736] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5434.618790] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5438.882216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5439.018661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.946708] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5443.377882] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5458.402234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.403414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5458.405717] Lustre: Skipped 2 previous similar messages [ 5458.408682] Lustre: Skipped 2 previous similar messages [ 5459.843402] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 5462.865633] Lustre: server umount lustre-MDT0000 complete [ 5463.520487] LustreError: 320715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.525568] LustreError: 320715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5464.068146] LustreError: 320678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076583 with bad export cookie 8152933640964842504 [ 5464.071587] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.074227] LustreError: 320678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5464.955811] LustreError: 320691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.962721] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 5468.639883] LustreError: 323186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.640171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.649372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5474.783964] LustreError: 320691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.784348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5474.788536] LustreError: 320691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5474.790392] Lustre: Skipped 2 previous similar messages [ 5478.367134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5478.440750] Lustre: server umount lustre-MDT0001 complete [ 5480.624706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5483.471021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.119578] LustreError: 320008:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000953cd7d x1869691262886016/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5489.253457] LustreError: 322528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.259074] LustreError: 322528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5489.279844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.300735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5490.693762] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5493.620609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5493.746658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5495.156180] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5497.609048] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5503.973791] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5507.329156] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5509.107928] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5509.640204] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5514.211390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.211432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5514.216215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5514.220907] Lustre: Skipped 2 previous similar messages [ 5517.076551] Lustre: server umount lustre-OST0000 complete [ 5519.329792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5519.330726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.336636] Lustre: Skipped 2 previous similar messages [ 5524.514527] Lustre: server umount lustre-MDT0000 complete [ 5525.788413] LustreError: 320677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076644 with bad export cookie 8152933640964849623 [ 5525.790904] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.792311] LustreError: 320677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5525.912403] Lustre: server umount lustre-MDT0001 complete [ 5529.806872] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5530.838324] Key type lgssc unregistered [ 5530.971348] LNet: 326977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5530.973906] LNetError: 326977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5530.986301] LNet: Removed LNI 192.168.201.152@tcp [ 5531.312603] Key type .llcrypt unregistered [ 5531.313809] Key type ._llcrypt unregistered [ 5540.499295] Key type ._llcrypt registered [ 5540.500307] Key type .llcrypt registered [ 5540.542036] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5540.971931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5540.993847] alg: No test for adler32 (adler32-zlib) [ 5541.868560] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5541.969865] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5543.551157] Key type lgssc registered [ 5543.947705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5546.583209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5546.589836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.756541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.126027] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5551.908451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.031760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5553.315123] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5555.754625] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5557.516086] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5562.336815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5562.337128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.337841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.348117] Lustre: Skipped 1 previous similar message [ 5563.360301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5563.362771] Lustre: Skipped 2 previous similar messages [ 5564.640145] Lustre: server umount lustre-MDT0000 complete [ 5565.935216] LustreError: 328398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076684 with bad export cookie 6841513288569832292 [ 5565.939668] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.086434] Lustre: server umount lustre-MDT0001 complete [ 5570.969341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.075495] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5571.088878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5571.178603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.630974] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5575.594475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.628321] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5575.640474] Lustre: 330956:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5575.746185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5577.150536] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5579.663754] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5581.547911] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5586.210721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5586.281156] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5586.284502] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5586.288403] LustreError: 332841:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5586.290833] LustreError: 332841:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5586.293089] LustreError: 332841:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5586.318345] Lustre: server umount lustre-OST0000 complete [ 5586.319607] LustreError: 332841:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5591.266377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5591.355414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5591.429507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.435087] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5595.895190] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5598.549903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5598.589173] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5598.616693] Lustre: lustre-OST0001: new disk, initializing [ 5598.618652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5598.621764] Lustre: 334369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5600.528420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5600.532698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5600.566292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5600.592793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5600.721705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5603.161727] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5605.856242] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.858212] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.859151] LustreError: Skipped 1 previous similar message [ 5605.860411] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5605.863207] Lustre: Skipped 1 previous similar message [ 5606.142419] Lustre: server umount lustre-OST0001 complete [ 5610.979875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5610.981377] LustreError: 333306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.982243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.983946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5610.983953] Lustre: Skipped 1 previous similar message [ 5610.987987] LustreError: 333306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5610.992292] Lustre: Skipped 1 previous similar message [ 5613.390586] Lustre: server umount lustre-OST0000 complete [ 5616.096615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.101244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5616.103142] Lustre: Skipped 2 previous similar messages [ 5620.763339] Lustre: server umount lustre-MDT0000 complete [ 5621.216116] LustreError: 330969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.223050] LustreError: 330969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5622.039354] LustreError: 331697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076741 with bad export cookie 6841513288569832831 [ 5622.042511] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.042990] LustreError: 331697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5622.167683] Lustre: server umount lustre-MDT0001 complete [ 5625.937690] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5626.981857] Key type lgssc unregistered [ 5627.109405] LNet: 336329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5627.111891] LNetError: 336329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5627.123321] LNet: Removed LNI 192.168.201.152@tcp [ 5627.451856] Key type .llcrypt unregistered [ 5627.453035] Key type ._llcrypt unregistered [ 5658.072501] Key type ._llcrypt registered [ 5658.073714] Key type .llcrypt registered [ 5658.114966] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5664.283687] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5664.722745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5664.734399] alg: No test for adler32 (adler32-zlib) [ 5665.586349] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5665.674944] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5667.263154] Key type lgssc registered [ 5667.634637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5679.166415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5679.194854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.296804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5680.311520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5680.353713] Lustre: lustre-MDT0000: new disk, initializing [ 5680.378678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.384837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5681.892077] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5686.111611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.151143] Lustre: 341231:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5686.173242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5686.175104] Lustre: Skipped 1 previous similar message [ 5686.212077] Lustre: lustre-MDT0001: new disk, initializing [ 5686.245070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5686.258258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5686.261701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5687.683502] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5691.434994] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5693.312304] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5695.707891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.829326] Lustre: lustre-OST0000: new disk, initializing [ 5695.831137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5695.833786] Lustre: 343198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5695.866593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.858844] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5701.589115] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5703.177088] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5704.704278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5704.708756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5704.750317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5705.288193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5706.722833] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5706.807538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5709.793317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5709.796924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.801567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5709.803445] Lustre: Skipped 1 previous similar message [ 5713.554614] Lustre: server umount lustre-OST0000 complete [ 5714.912346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.913113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5714.917844] Lustre: Skipped 2 previous similar messages [ 5720.032195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.034646] Lustre: Skipped 2 previous similar messages [ 5725.152347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.154850] Lustre: Skipped 1 previous similar message [ 5729.247134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5729.248341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5729.252025] Lustre: Skipped 1 previous similar message [ 5729.307933] Lustre: server umount lustre-MDT0000 complete [ 5730.642607] LustreError: 341225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076849 with bad export cookie 5882263541505376922 [ 5730.647800] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5730.647955] LustreError: 341225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5730.786867] Lustre: server umount lustre-MDT0001 complete [ 5733.579084] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:07:31 (1783076851) [ 5735.977444] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:07:34 (1783076854) [ 5738.953816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.163765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5740.609734] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5743.541809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.677255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5745.114524] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5747.607512] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5749.500607] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5751.842221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5751.977881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.943605] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5756.410404] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5770.467934] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5771.045730] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5771.635127] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5772.188826] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5772.836249] Lustre: Failing over lustre-MDT0000 [ 5773.096411] Lustre: server umount lustre-MDT0000 complete [ 5774.304060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5774.306929] LustreError: Skipped 1 previous similar message [ 5774.309047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.312674] LustreError: 345964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.316989] LustreError: 345964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5774.817987] LustreError: 348456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.823201] LustreError: 348456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5777.281743] LustreError: 347377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.936272] LustreError: 345965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.943187] LustreError: 345965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5785.056239] LustreError: 345983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.060815] LustreError: 345983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5786.073292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.111879] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.210850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5786.235744] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5787.515488] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5787.542518] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5791.717071] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5791.727838] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5791.747476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5793.511070] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5794.041096] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5795.876511] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5796.554608] Lustre: Failing over lustre-MDT0000 [ 5796.716568] Lustre: server umount lustre-MDT0000 complete [ 5796.832944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5796.833638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.834518] LustreError: 345959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.843385] Lustre: Skipped 3 previous similar messages [ 5809.638734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.681347] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5809.814538] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5811.135697] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5813.116104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5815.269372] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5815.271195] Lustre: Skipped 2 previous similar messages [ 5815.279600] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5815.297439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5817.102290] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.681256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5820.387815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5820.388427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.389739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.397374] Lustre: Skipped 2 previous similar messages [ 5826.071848] Lustre: server umount lustre-MDT0000 complete [ 5827.283062] LustreError: 345945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783076946 with bad export cookie 5882263541505379603 [ 5827.285837] LustreError: 345945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5827.286682] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.392640] Lustre: server umount lustre-MDT0001 complete [ 5838.809811] Lustre: server umount lustre-OST0000 complete [ 5843.889639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.059260] LustreError: 352314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.064576] LustreError: 352314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5844.082486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.085546] Lustre: Skipped 1 previous similar message [ 5845.406427] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5848.178938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.631832] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5852.040882] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5853.821485] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5856.029197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5858.016518] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5860.426404] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5869.477245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5873.631763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.634087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.638089] Lustre: Skipped 1 previous similar message [ 5873.639865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5873.641481] Lustre: Skipped 5 previous similar messages [ 5878.356497] Lustre: server umount lustre-OST0000 complete [ 5884.383959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5884.386014] LustreError: Skipped 1 previous similar message [ 5885.785673] Lustre: server umount lustre-MDT0000 complete [ 5887.062349] LustreError: 352295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077006 with bad export cookie 5882263541505380324 [ 5887.065864] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5887.067133] LustreError: 352295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5887.195192] Lustre: server umount lustre-MDT0001 complete [ 5890.703360] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5891.718932] Key type lgssc unregistered [ 5891.844377] LNet: 355864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5891.848321] LNetError: 355864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5891.859332] LNet: Removed LNI 192.168.201.152@tcp [ 5892.180182] Key type .llcrypt unregistered [ 5892.181280] Key type ._llcrypt unregistered [ 5901.818915] Key type ._llcrypt registered [ 5901.820063] Key type .llcrypt registered [ 5901.853689] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5902.322341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5902.332105] alg: No test for adler32 (adler32-zlib) [ 5903.181852] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 5903.272813] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5904.847151] Key type lgssc registered [ 5905.208381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5907.697257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5907.702847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5908.844394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.140631] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5912.991520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.124201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5914.509324] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5916.944848] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5918.722175] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5920.942610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5921.056773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.011815] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5925.343192] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5934.379276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5943.776361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5943.779325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.783789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.800431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.802457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.806305] Lustre: Skipped 1 previous similar message [ 5945.941865] Lustre: server umount lustre-MDT0000 complete [ 5947.109237] LustreError: 357337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077066 with bad export cookie 5967886086832817001 [ 5947.113146] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.114624] LustreError: 357337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5947.223282] Lustre: server umount lustre-MDT0001 complete [ 5958.746476] Lustre: server umount lustre-OST0000 complete [ 5963.129902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.282880] LustreError: 361086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.290888] LustreError: 361086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5963.306099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.697437] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5967.606996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.727057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.169728] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5971.781418] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5973.638030] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5975.996178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.120129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.119189] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5980.610153] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5989.418092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5998.559881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5998.562761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.567729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5998.569237] Lustre: Skipped 2 previous similar messages [ 6004.239939] Lustre: server umount lustre-MDT0000 complete [ 6004.704461] LustreError: 361105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.709848] LustreError: 361105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6005.399220] LustreError: 362928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077124 with bad export cookie 5967886086832818100 [ 6005.403194] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.404744] LustreError: 362928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6005.506911] Lustre: server umount lustre-MDT0001 complete [ 6016.988262] Lustre: server umount lustre-OST0000 complete [ 6021.867713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.015995] LustreError: 365016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.037803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.338883] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6026.045434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.486513] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6029.919906] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6031.752849] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6033.962393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6036.114316] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6038.548360] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6047.402066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6051.810177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6051.812472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.815564] Lustre: Skipped 2 previous similar messages [ 6051.817441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6051.818976] Lustre: Skipped 3 previous similar messages [ 6056.462980] Lustre: server umount lustre-OST0000 complete [ 6057.952446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6057.953275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6057.956072] Lustre: Skipped 2 previous similar messages [ 6057.960947] Lustre: Skipped 3 previous similar messages [ 6068.192523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6068.194015] Lustre: Skipped 3 previous similar messages [ 6072.287128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6072.350032] Lustre: server umount lustre-MDT0000 complete [ 6073.313039] LustreError: 365016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.318105] LustreError: 365016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6073.606127] LustreError: 364996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077192 with bad export cookie 5967886086832819108 [ 6073.609960] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6073.611372] LustreError: 364996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6073.727823] Lustre: server umount lustre-MDT0001 complete [ 6077.476176] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6078.551503] Key type lgssc unregistered [ 6078.684424] LNet: 368570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6078.687692] LNetError: 368570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6078.693332] LNet: Removed LNI 192.168.201.152@tcp [ 6079.014607] Key type .llcrypt unregistered [ 6079.016090] Key type ._llcrypt unregistered [ 6088.352176] Key type ._llcrypt registered [ 6088.353457] Key type .llcrypt registered [ 6088.401450] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6088.903050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6088.919447] alg: No test for adler32 (adler32-zlib) [ 6089.789215] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6089.881488] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6091.471200] Key type lgssc registered [ 6091.905555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6094.648418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6094.654510] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.657243] LustreError: 369986:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6094.664880] LustreError: 369986:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6094.666933] LustreError: 369986:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6094.669076] LustreError: 369986:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6094.671297] LustreError: 369986:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6097.207978] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6097.211160] LustreError: 370383:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6097.218911] LustreError: 370383:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6097.221519] LustreError: 370383:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6097.224100] LustreError: 370383:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6097.226964] LustreError: 370383:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6103.584951] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6104.593362] Key type lgssc unregistered [ 6104.721326] LNet: 371384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.724505] LNetError: 371384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.731261] LNet: Removed LNI 192.168.201.152@tcp [ 6105.039408] Key type .llcrypt unregistered [ 6105.040506] Key type ._llcrypt unregistered [ 6110.919256] Key type ._llcrypt registered [ 6110.920445] Key type .llcrypt registered [ 6110.962389] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6117.350295] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6117.693580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.744968] alg: No test for adler32 (adler32-zlib) [ 6118.603333] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6118.694793] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6120.271204] Key type lgssc registered [ 6120.651828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6131.778125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6131.784489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.872442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6132.881962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6132.918357] Lustre: lustre-MDT0000: new disk, initializing [ 6132.939432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.945724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6134.306098] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6138.430949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.474886] Lustre: 374512:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6138.499179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6138.501080] Lustre: Skipped 1 previous similar message [ 6138.541662] Lustre: lustre-MDT0001: new disk, initializing [ 6138.576292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6138.589252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6138.592623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6139.956036] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6143.715102] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6145.558279] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6147.876168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6148.012316] Lustre: lustre-OST0000: new disk, initializing [ 6148.014734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6148.017441] Lustre: 376480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6148.057887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.062214] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6153.737050] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6155.396389] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6155.766406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6155.770205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6155.797632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6156.490087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6157.839851] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6157.910572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6160.866731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.866993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6160.868205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.870922] Lustre: Skipped 1 previous similar message [ 6160.874670] LustreError: Skipped 1 previous similar message [ 6164.635827] Lustre: server umount lustre-OST0000 complete [ 6165.985181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6165.986060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.988976] Lustre: Skipped 1 previous similar message [ 6165.990890] Lustre: Skipped 1 previous similar message [ 6171.104335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6171.107137] Lustre: Skipped 2 previous similar messages [ 6176.224183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6176.226959] Lustre: Skipped 1 previous similar message [ 6180.319102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6180.393807] Lustre: server umount lustre-MDT0000 complete [ 6181.735834] LustreError: 375309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077300 with bad export cookie 17870543297498354917 [ 6181.739309] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6181.740222] LustreError: 375309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6181.879291] Lustre: server umount lustre-MDT0001 complete [ 6184.810875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6185.327268] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:15:03 (1783077303) [ 6187.443277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.596850] LustreError: 378944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.603928] LustreError: 378944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6187.621412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.908368] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6191.623842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.746642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6193.050119] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6195.301614] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6196.992716] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6199.046436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6200.854557] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6203.017118] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6205.128114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.182402] Lustre: lustre-OST0001: new disk, initializing [ 6205.183856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6205.185995] Lustre: 381778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6205.212226] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6205.214776] Lustre: Skipped 1 previous similar message [ 6206.957943] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6210.431729] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6210.560764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6210.564102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6210.594098] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6213.218666] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.222192] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6213.223143] Lustre: Skipped 1 previous similar message [ 6218.301991] Lustre: server umount lustre-OST0001 complete [ 6218.724918] LustreError: 381191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.729503] LustreError: 381191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6223.844032] LustreError: 381191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.850127] LustreError: 381191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6228.962453] LustreError: 380763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.969483] LustreError: 380763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6232.031885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6232.034253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.039134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6232.041147] Lustre: Skipped 2 previous similar messages [ 6234.085885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6234.086869] LustreError: 382587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.095524] LustreError: 382587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6236.431373] Lustre: server umount lustre-OST0000 complete [ 6238.175873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6238.178623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.182954] Lustre: Skipped 1 previous similar message [ 6244.004184] Lustre: server umount lustre-MDT0000 complete [ 6244.321087] LustreError: 378944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.326632] LustreError: 378944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6245.310280] LustreError: 378926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077364 with bad export cookie 17870543297498355785 [ 6245.313455] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6245.314479] LustreError: 378926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6251.504521] Lustre: server umount lustre-MDT0001 complete [ 6255.121250] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6256.190033] Key type lgssc unregistered [ 6256.324360] LNet: 383763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6256.326637] LNetError: 383763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6256.335408] LNet: Removed LNI 192.168.201.152@tcp [ 6256.659270] Key type .llcrypt unregistered [ 6256.660333] Key type ._llcrypt unregistered [ 6274.795432] Key type ._llcrypt registered [ 6274.796445] Key type .llcrypt registered [ 6274.836772] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6275.157489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.166606] alg: No test for adler32 (adler32-zlib) [ 6276.019104] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6276.108586] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6277.687151] Key type lgssc registered [ 6278.048680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6280.648884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6280.655658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.763795] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6281.770967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6281.832553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.185923] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6286.156276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.191853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6286.206269] Lustre: 386532:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6286.322063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6287.705816] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6290.233533] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6292.024383] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6296.543869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6296.546656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6296.550481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.568875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.569801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.688030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6302.690023] Lustre: Skipped 2 previous similar messages [ 6307.295145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6307.353505] Lustre: server umount lustre-MDT0000 complete [ 6308.587251] LustreError: 386526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077427 with bad export cookie 10991408135948093498 [ 6308.590284] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.592603] LustreError: 386526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.704124] Lustre: server umount lustre-MDT0001 complete [ 6311.689328] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:17:09 (1783077429) [ 6313.865367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.039325] LustreError: 389045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.043550] LustreError: 389045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6314.057383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.447654] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6318.194078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.314222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6319.696447] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6322.096264] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6323.949228] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6326.278621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6326.362203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6326.449507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.348242] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6330.694858] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6334.424258] Lustre: Permanently deactivating lustre-OST0000 [ 6334.430405] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6337.970047] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6338.954774] LustreError: 391982:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6338.957947] LustreError: 391982:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6338.960983] LustreError: 391982:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6339.591081] LustreError: 392031:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6339.595718] LustreError: 392031:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6346.318498] Lustre: server umount lustre-OST0000 complete [ 6347.237544] LustreError: 392280:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6347.240729] LustreError: 392280:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6349.279868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6349.282749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.286676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6355.936239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6355.938270] Lustre: Skipped 3 previous similar messages [ 6356.985188] Lustre: server umount lustre-MDT0000 complete [ 6358.262118] LustreError: 391736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077477 with bad export cookie 10991408135948094065 [ 6358.265183] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.266746] LustreError: 391736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6368.783516] Lustre: server umount lustre-MDT0001 complete [ 6371.620849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.937362] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6373.811270] LustreError: 393438:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6375.951909] Lustre: 393776:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.152@tcp [ 6376.654841] Lustre: server umount MGS complete [ 6380.477440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.624669] LustreError: 394498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.629911] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6380.634387] Lustre: Skipped 1 previous similar message [ 6380.653321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6381.991650] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6384.837131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.937988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6386.329044] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6388.584571] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6390.445059] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6391.222279] Lustre: Permanently reactivating lustre-OST0000 [ 6391.227328] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6391.229286] Lustre: Skipped 2 previous similar messages [ 6414.303247] LustreError: 396228:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6417.187707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.304520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6417.307830] Lustre: Skipped 1 previous similar message [ 6419.309714] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6421.718426] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6434.789260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.790109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.791608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6434.795808] LustreError: Skipped 1 previous similar message [ 6434.803438] Lustre: Skipped 2 previous similar messages [ 6435.466156] Lustre: server umount lustre-OST0000 complete [ 6438.882859] Lustre: server umount lustre-MDT0000 complete [ 6439.905069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.910348] LustreError: 395225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.915344] LustreError: 395225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6440.112388] LustreError: 394479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077559 with bad export cookie 10991408135948095101 [ 6440.114826] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.116167] LustreError: 394479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6440.230915] Lustre: server umount lustre-MDT0001 complete [ 6443.561472] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6444.598135] Key type lgssc unregistered [ 6444.725452] LNet: 399066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6444.727639] LNetError: 399066:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6444.738373] LNet: Removed LNI 192.168.201.152@tcp [ 6445.046610] Key type .llcrypt unregistered [ 6445.047703] Key type ._llcrypt unregistered [ 6451.221803] Key type ._llcrypt registered [ 6451.222932] Key type .llcrypt registered [ 6451.257819] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6457.183657] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6457.515764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6457.546062] alg: No test for adler32 (adler32-zlib) [ 6458.395705] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6458.490240] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6460.071138] Key type lgssc registered [ 6460.436160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6471.676321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6471.681361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.767534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6472.777642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6472.827254] Lustre: lustre-MDT0000: new disk, initializing [ 6472.852800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6472.858538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6474.151734] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6478.266931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.310992] Lustre: 402195:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6478.334316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6478.335933] Lustre: Skipped 1 previous similar message [ 6478.373545] Lustre: lustre-MDT0001: new disk, initializing [ 6478.404264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6478.416274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6478.419707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6479.787549] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6483.569174] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6485.430867] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6487.832931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6487.934486] Lustre: lustre-OST0000: new disk, initializing [ 6487.936484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6487.939089] Lustre: 404161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6487.965625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.837938] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6493.482364] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6494.931711] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6496.253391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6496.257441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6496.299951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6497.039624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6498.343248] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6498.411769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6501.346225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6501.346632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6501.352829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6501.354703] LustreError: Skipped 1 previous similar message [ 6505.170814] Lustre: server umount lustre-OST0000 complete [ 6506.464568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.465194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6506.469993] Lustre: Skipped 2 previous similar messages [ 6506.471760] Lustre: Skipped 1 previous similar message [ 6511.584121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6511.585859] Lustre: Skipped 1 previous similar message [ 6512.539846] Lustre: server umount lustre-MDT0000 complete [ 6513.739371] LustreError: 402990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077632 with bad export cookie 4027718424865327462 [ 6513.741948] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.742322] LustreError: 402990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6513.864785] Lustre: server umount lustre-MDT0001 complete [ 6516.620115] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6517.214199] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:20:35 (1783077635) [ 6519.491304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.678747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6521.055808] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6523.882487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.014261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6525.318671] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6527.712562] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6529.486228] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6531.700610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.698890] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6536.132043] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6536.784629] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6547.400437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6549.476490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.478488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.478870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6549.479558] Lustre: Skipped 1 previous similar message [ 6554.592891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6554.594570] Lustre: Skipped 1 previous similar message [ 6554.952613] Lustre: server umount lustre-OST0000 complete [ 6559.712902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6559.713063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.718151] Lustre: Skipped 1 previous similar message [ 6562.259547] Lustre: server umount lustre-MDT0000 complete [ 6563.468189] LustreError: 406610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077682 with bad export cookie 4027718424865328330 [ 6563.470164] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.471580] LustreError: 406610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6563.598204] Lustre: server umount lustre-MDT0001 complete [ 6567.000320] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6568.021941] Key type lgssc unregistered [ 6568.150324] LNet: 410371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6568.152568] LNetError: 410371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6568.158307] LNet: Removed LNI 192.168.201.152@tcp [ 6568.453304] Key type .llcrypt unregistered [ 6568.454564] Key type ._llcrypt unregistered [ 6576.259194] Key type ._llcrypt registered [ 6576.260443] Key type .llcrypt registered [ 6576.294894] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6576.384368] Key type .llcrypt unregistered [ 6576.385389] Key type ._llcrypt unregistered [ 6583.468880] Key type ._llcrypt registered [ 6583.469986] Key type .llcrypt registered [ 6583.510252] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6584.091808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6584.102468] alg: No test for adler32 (adler32-zlib) [ 6584.974486] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6585.069722] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6586.655186] Key type lgssc registered [ 6587.114645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6589.684722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6589.690598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.826121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.167624] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6595.781731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.934986] LustreError: 412553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.938053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.940318] LustreError: 412553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6597.734694] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6600.032678] LustreError: 412558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.037386] LustreError: 412558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6600.037892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6600.142255] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6602.312238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.428175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.758741] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6612.961417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6612.963644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.967623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6612.970366] Lustre: Skipped 1 previous similar message [ 6617.033684] Lustre: server umount lustre-OST0000 complete [ 6622.177323] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6622.179425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.182791] Lustre: Skipped 1 previous similar message [ 6622.185252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.200064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.200616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.341056] Lustre: server umount lustre-MDT0000 complete [ 6625.479262] LustreError: 412541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077744 with bad export cookie 18220011503843656147 [ 6625.482535] LustreError: 412541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6625.482991] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.607215] Lustre: server umount lustre-MDT0001 complete [ 6628.991423] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6629.986698] Key type lgssc unregistered [ 6630.115461] LNet: 415709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6630.117771] LNetError: 415709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6630.125369] LNet: Removed LNI 192.168.201.152@tcp [ 6630.452083] Key type .llcrypt unregistered [ 6630.453943] Key type ._llcrypt unregistered [ 6639.360143] Key type ._llcrypt registered [ 6639.361022] Key type .llcrypt registered [ 6639.393526] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6639.685790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6639.711109] alg: No test for adler32 (adler32-zlib) [ 6640.549129] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6640.628361] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6642.199140] Key type lgssc registered [ 6642.531904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6644.917867] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6644.923875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6660.383333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6665.503299] LustreError: 417075:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 6692.127145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.011673] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6696.446191] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6698.621863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.810267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.834812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6700.196772] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6702.988285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.105178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6704.316219] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6706.537685] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6708.247904] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6711.266293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.267480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6711.272298] Lustre: Skipped 1 previous similar message [ 6711.274966] Lustre: Skipped 1 previous similar message [ 6716.386522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6716.388377] Lustre: Skipped 1 previous similar message [ 6716.623890] Lustre: server umount lustre-OST0000 complete [ 6718.431897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6718.435222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.441040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.505207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6721.506270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.986685] Lustre: server umount lustre-MDT0000 complete [ 6725.106988] LustreError: 417081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077844 with bad export cookie 4234995093958918580 [ 6725.110568] LustreError: 417081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6725.110772] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6725.214028] Lustre: server umount lustre-MDT0001 complete [ 6728.615755] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6729.658389] Key type lgssc unregistered [ 6729.800374] LNet: 420643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6729.802696] LNetError: 420643:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6729.809459] LNet: Removed LNI 192.168.201.152@tcp [ 6730.154904] Key type .llcrypt unregistered [ 6730.155944] Key type ._llcrypt unregistered [ 6738.935685] Key type ._llcrypt registered [ 6738.936686] Key type .llcrypt registered [ 6738.978263] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6739.374410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6739.434086] alg: No test for adler32 (adler32-zlib) [ 6740.289579] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6740.378133] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6741.959137] Key type lgssc registered [ 6742.306863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6744.886833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6744.893154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.047435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.496109] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6750.398875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6750.520656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.879203] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6754.103049] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6755.946495] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6758.431832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6758.573405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6760.697369] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6763.151550] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6771.877467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6776.288388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6776.290547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.295245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6777.313141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6777.314170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.315121] Lustre: Skipped 1 previous similar message [ 6777.318970] Lustre: Skipped 1 previous similar message [ 6778.757285] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 6780.756019] Lustre: server umount lustre-MDT0000 complete [ 6781.479487] LustreError: 423878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.52@tcp arrived at 1783077900 with bad export cookie 10854118746139846702 [ 6781.484187] LustreError: 423878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6782.094297] LustreError: 422016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783077901 with bad export cookie 10854118746139846051 [ 6782.098156] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6782.431953] LustreError: 422774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.432468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.436649] LustreError: 422774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6782.440274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6787.552237] LustreError: 422035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.552679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6787.561035] LustreError: 422035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6788.250695] Lustre: server umount lustre-OST0000 complete [ 6790.539816] Lustre: server umount lustre-MDT0001 complete [ 6793.875756] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6794.918991] Key type lgssc unregistered [ 6795.039471] LNet: 425736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6795.041571] LNetError: 425736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6795.051357] LNet: Removed LNI 192.168.201.152@tcp [ 6795.352977] Key type .llcrypt unregistered [ 6795.354080] Key type ._llcrypt unregistered [ 6804.027378] Key type ._llcrypt registered [ 6804.028792] Key type .llcrypt registered [ 6804.068170] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6804.441826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6804.473073] alg: No test for adler32 (adler32-zlib) [ 6805.351655] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6805.457801] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6807.063100] Key type lgssc registered [ 6807.468114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6810.766883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6810.780060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.964397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.486428] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6816.842630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.985373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6818.714508] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6821.781589] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6824.056125] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6826.954383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.100655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6829.656878] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6833.003542] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6837.357025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6842.855872] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6842.856926] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.862451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6842.869508] Lustre: Skipped 1 previous similar message [ 6843.789297] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 6843.793216] Lustre: Skipped 1 previous similar message [ 6845.731452] Lustre: server umount lustre-MDT0001 complete [ 6847.969829] LustreError: 427128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.970339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6847.970667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.972085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.979608] LustreError: 427128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6853.089915] LustreError: 427129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.090341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6853.610149] Lustre: server umount lustre-OST0000 complete [ 6855.665974] Lustre: server umount lustre-MDT0000 complete [ 6861.049896] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6862.391696] Key type lgssc unregistered [ 6862.556802] LNet: 430808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6862.561222] LNetError: 430808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6862.572627] LNet: Removed LNI 192.168.201.152@tcp [ 6863.047556] Key type .llcrypt unregistered [ 6863.049939] Key type ._llcrypt unregistered [ 6873.206915] Key type ._llcrypt registered [ 6873.208335] Key type .llcrypt registered [ 6873.259713] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6880.570181] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6880.994668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6881.041263] alg: No test for adler32 (adler32-zlib) [ 6881.893484] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6881.990224] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6883.575141] Key type lgssc registered [ 6883.918974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6895.573855] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6900.430648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6900.436211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.536441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6901.549119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6901.599843] Lustre: lustre-MDT0000: new disk, initializing [ 6901.621877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.627740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6902.998405] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6908.543113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6908.666217] Lustre: lustre-OST0000: new disk, initializing [ 6908.667967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6908.669659] Lustre: Skipped 1 previous similar message [ 6908.671485] Lustre: 435812:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6908.699990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6909.702498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6909.705317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6909.713791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6910.680807] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6914.314678] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6919.754100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6919.799689] Lustre: 434795:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6919.835921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6919.875484] Lustre: lustre-MDT0001: new disk, initializing [ 6919.924270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6919.942500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6919.946902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6921.276324] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6924.819423] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6924.890065] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6926.189426] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6929.325236] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 6933.893167] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 6933.984602] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6933.985396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.990466] Lustre: Skipped 1 previous similar message [ 6939.004672] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 6939.007217] Lustre: Skipped 2 previous similar messages [ 6944.123745] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 6944.125845] Lustre: Skipped 2 previous similar messages [ 6945.247110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6945.304458] Lustre: server umount lustre-MDT0001 complete [ 6954.465651] LustreError: 434799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.472430] LustreError: 434799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6959.586053] LustreError: 434804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.586144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.594815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6959.596495] Lustre: Skipped 2 previous similar messages [ 6963.664356] Lustre: server umount lustre-OST0000 complete [ 6964.705156] LustreError: 437149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.073520] Lustre: server umount lustre-MDT0000 complete [ 6975.352769] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6976.388986] Key type lgssc unregistered [ 6976.508433] LNet: 439578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6976.510546] LNetError: 439578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6976.520404] LNet: Removed LNI 192.168.201.152@tcp [ 6976.822527] Key type .llcrypt unregistered [ 6976.824229] Key type ._llcrypt unregistered [ 6988.914632] Key type ._llcrypt registered [ 6988.916716] Key type .llcrypt registered [ 6988.985996] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6989.165309] Key type .llcrypt unregistered [ 6989.167389] Key type ._llcrypt unregistered [ 6995.645072] Key type ._llcrypt registered [ 6995.646251] Key type .llcrypt registered [ 6995.691300] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6996.091520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6996.101296] alg: No test for adler32 (adler32-zlib) [ 6996.981763] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 6997.104040] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6998.695200] Key type lgssc registered [ 6999.195192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7009.027785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7009.034916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.138245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7010.148352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7010.191256] Lustre: lustre-MDT0000: new disk, initializing [ 7010.212176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7010.217275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7011.870603] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7016.450714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7016.491369] Lustre: 442926:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7016.514076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7016.515747] Lustre: Skipped 1 previous similar message [ 7016.556934] Lustre: lustre-MDT0001: new disk, initializing [ 7016.591511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7016.602731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7016.606764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7017.959578] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7021.958297] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7023.890253] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7026.186711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7026.354168] Lustre: lustre-OST0000: new disk, initializing [ 7026.356528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7026.360247] Lustre: 444896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7026.397181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7028.738759] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7033.200623] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7035.908807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7035.914585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7035.958190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7042.528793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7042.533791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.542710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.115721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.116721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.123785] Lustre: Skipped 1 previous similar message [ 7046.129190] Lustre: Skipped 1 previous similar message [ 7047.481228] Lustre: server umount lustre-MDT0000 complete [ 7048.784375] LustreError: 442920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783078167 with bad export cookie 7567833816581671646 [ 7048.787924] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.789455] LustreError: 442920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7048.944850] Lustre: server umount lustre-MDT0001 complete [ 7060.464305] Lustre: server umount lustre-OST0000 complete [ 7064.580822] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:29:42 (1783078182) [ 7067.415070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.691692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.262874] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7072.738398] LustreError: 446926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.747736] LustreError: 446926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7073.089605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.289297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7075.069752] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7078.494242] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7080.879371] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7097.062385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7097.182749] Lustre: Found index 0 for lustre-OST0000, updating log [ 7097.250575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.791446] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7102.867923] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7113.640064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7123.939452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7123.942215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.944333] LustreError: Skipped 1 previous similar message [ 7123.951792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7123.954339] Lustre: Skipped 2 previous similar messages [ 7126.134647] Lustre: server umount lustre-OST0000 complete [ 7139.808646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.813322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.820465] Lustre: Skipped 1 previous similar message [ 7139.823737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.829136] Lustre: Skipped 2 previous similar messages [ 7145.496811] Lustre: server umount lustre-MDT0000 complete [ 7146.464922] LustreError: 446926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.471871] LustreError: 446926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7147.248711] LustreError: 447660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783078266 with bad export cookie 7567833816581673018 [ 7147.254351] LustreError: 447660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7147.256325] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.381121] Lustre: server umount lustre-MDT0001 complete [ 7150.671018] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:31:08 (1783078268) [ 7158.879355] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [ 7734.557355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7734.780941] LustreError: 453347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.825933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.597660] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7739.873649] LustreError: 453348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.590468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.726297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7742.296437] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7745.581201] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7747.889976] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7763.746849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.853494] Lustre: Found index 0 for lustre-OST0000, updating log [ 7763.962757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.928835] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7768.755649] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7773.154196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.155053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7773.158686] Lustre: Skipped 1 previous similar message [ 7773.164343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7773.167458] Lustre: Skipped 2 previous similar messages [ 7775.723116] Lustre: server umount lustre-OST0000 complete [ 7778.336927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.337872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.340155] Lustre: Skipped 1 previous similar message [ 7779.361901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7779.365766] Lustre: Skipped 2 previous similar messages [ 7781.409413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.413496] Lustre: Skipped 3 previous similar messages [ 7783.532876] Lustre: server umount lustre-MDT0000 complete [ 7784.480983] LustreError: 454587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.489020] LustreError: 454587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7785.324970] LustreError: 453329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783078904 with bad export cookie 7567833816581674096 [ 7785.331530] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.332751] LustreError: 453329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7785.484265] Lustre: server umount lustre-MDT0001 complete [ 7786.591308] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [ 7804.744728] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 7808.607212] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7821.507969] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:42:19 (1783078939) [ 7827.915422] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7829.224634] Key type lgssc unregistered [ 7829.400689] LNet: 460002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7829.405508] LNetError: 460002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.949517] LNet: Removed LNI 192.168.201.152@tcp [ 7895.407424] Key type .llcrypt unregistered [ 7895.409518] Key type ._llcrypt unregistered [ 7903.255813] Key type ._llcrypt registered [ 7903.257931] Key type .llcrypt registered [ 7903.314897] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7903.728352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7903.794352] alg: No test for adler32 (adler32-zlib) [ 7904.672880] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 7904.773476] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 7906.375169] Key type lgssc registered [ 7906.988072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7908.951986] LNet: Added LNI 192.168.201.152@tcp100 [8/256/0/180] [ 7909.352254] LNet: Added LNI 192.168.201.152@tcp101 [8/256/0/180] [ 7909.758919] LNet: Added LNI 192.168.201.152@tcp102 [8/256/0/180] [ 7910.163342] LNet: Added LNI 192.168.201.152@tcp103 [8/256/0/180] [ 7910.578799] LNet: Added LNI 192.168.201.152@tcp104 [8/256/0/180] [ 7911.007942] LNet: Added LNI 192.168.201.152@tcp105 [8/256/0/180] [ 7911.431506] LNet: Added LNI 192.168.201.152@tcp106 [8/256/0/180] [ 7911.806497] LNet: Added LNI 192.168.201.152@tcp107 [8/256/0/180] [ 7912.217605] LNet: Added LNI 192.168.201.152@tcp108 [8/256/0/180] [ 7912.638853] LNet: Added LNI 192.168.201.152@tcp109 [8/256/0/180] [ 7913.057174] LNet: Added LNI 192.168.201.152@tcp110 [8/256/0/180] [ 7913.455650] LNet: Added LNI 192.168.201.152@tcp111 [8/256/0/180] [ 7913.863685] LNet: Added LNI 192.168.201.152@tcp112 [8/256/0/180] [ 7914.294603] LNet: Added LNI 192.168.201.152@tcp113 [8/256/0/180] [ 7914.710989] LNet: Added LNI 192.168.201.152@tcp114 [8/256/0/180] [ 7915.126047] LNet: Added LNI 192.168.201.152@tcp115 [8/256/0/180] [ 7915.536907] LNet: Added LNI 192.168.201.152@tcp116 [8/256/0/180] [ 7915.961317] LNet: Added LNI 192.168.201.152@tcp117 [8/256/0/180] [ 7916.388559] LNet: Added LNI 192.168.201.152@tcp118 [8/256/0/180] [ 7916.782842] LNet: Added LNI 192.168.201.152@tcp119 [8/256/0/180] [ 7917.193058] LNet: Added LNI 192.168.201.152@tcp120 [8/256/0/180] [ 7917.595347] LNet: Added LNI 192.168.201.152@tcp121 [8/256/0/180] [ 7917.990302] LNet: Added LNI 192.168.201.152@tcp122 [8/256/0/180] [ 7918.285851] LNet: Added LNI 192.168.201.152@tcp123 [8/256/0/180] [ 7918.592898] LNet: Added LNI 192.168.201.152@tcp124 [8/256/0/180] [ 7919.015572] LNet: Added LNI 192.168.201.152@tcp125 [8/256/0/180] [ 7919.436220] LNet: Added LNI 192.168.201.152@tcp126 [8/256/0/180] [ 7919.821153] LNet: Added LNI 192.168.201.152@tcp127 [8/256/0/180] [ 7920.224529] LNet: Added LNI 192.168.201.152@tcp128 [8/256/0/180] [ 7920.608638] LNet: Added LNI 192.168.201.152@tcp129 [8/256/0/180] [ 7920.987245] LNet: Added LNI 192.168.201.152@tcp130 [8/256/0/180] [ 7921.373577] LNet: Added LNI 192.168.201.152@tcp131 [8/256/0/180] [ 7921.784354] LNet: Added LNI 192.168.201.152@tcp132 [8/256/0/180] [ 7922.194864] LNet: Added LNI 192.168.201.152@tcp133 [8/256/0/180] [ 7922.600038] LNet: Added LNI 192.168.201.152@tcp134 [8/256/0/180] [ 7922.997327] LNet: Added LNI 192.168.201.152@tcp135 [8/256/0/180] [ 7925.798845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7925.805436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.844144] Lustre: 463001:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [ 7926.879022] LNetError: 460462:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [ 7926.882421] LNetError: Protocol error connecting to 192.168.201.152@tcp118 at host 192.168.201.152:988: is it running a compatible version of Lustre? [ 7926.885546] LNetError: Skipped 2 previous similar messages [ 7926.912454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7926.922599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7926.961070] Lustre: lustre-MDT0000: new disk, initializing [ 7926.982967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7926.989591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7927.327240] LNet: 3 local NIs in recovery (showing 3): 192.168.201.152@tcp125, 192.168.201.152@tcp128, 192.168.201.152@tcp129 [ 7928.604288] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7929.377431] LNetError: 463031:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [ 7929.380398] LNetError: 463031:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [ 7929.382651] LNetError: Protocol error connecting to 192.168.201.152@tcp125 at host 192.168.201.152:988: is it running a compatible version of Lustre? [ 7929.385982] LNetError: Skipped 17 previous similar messages [ 7933.473024] LNetError: 460462:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [ 7933.478170] LNetError: Protocol error connecting to 192.168.201.152@tcp125 at host 192.168.201.152:988: is it running a compatible version of Lustre? [ 7933.559693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.627400] Lustre: 463016:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7933.661690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7933.663944] Lustre: Skipped 1 previous similar message [ 7933.700985] Lustre: lustre-MDT0001: new disk, initializing [ 7933.715128] Lustre: 463819:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [ 7933.718878] Lustre: 463819:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7933.776245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7933.795780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7933.801713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7935.348219] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7936.378589] Lustre: 464182:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [ 7936.384815] Lustre: 464182:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7940.053507] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7941.664664] LNetError: 460465:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [ 7941.670931] LNetError: Protocol error connecting to 192.168.201.152@tcp125 at host 192.168.201.152:988: is it running a compatible version of Lustre? [ 7942.403463] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7945.473422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7945.625498] Lustre: lustre-OST0000: new disk, initializing [ 7945.629089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7945.634393] Lustre: 464989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7945.691250] Lustre: 464994:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [ 7945.696087] Lustre: 464994:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7945.719483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7948.101362] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7950.842921] Lustre: 465537:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [ 7950.850038] Lustre: 465537:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7950.873527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7950.878947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7950.931534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7952.549512] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7955.941948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.942016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7955.951053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7955.958979] Lustre: Skipped 1 previous similar message [ 7957.028364] LNetError: 463022:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [ 7957.034523] LNetError: Protocol error connecting to 192.168.201.152@tcp112 at host 192.168.201.152:988: is it running a compatible version of Lustre? [ 7957.603405] Lustre: server umount lustre-OST0000 complete [ 7959.520369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7959.524603] LustreError: Skipped 1 previous similar message [ 7959.526975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.532399] Lustre: Skipped 1 previous similar message [ 7959.534745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7961.056877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7961.058228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.424933] Lustre: server umount lustre-MDT0000 complete [ 7966.244878] LNetError: 460465:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [ 7966.249472] LNetError: 460465:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7966.252972] LNetError: Protocol error connecting to 192.168.201.152@tcp106 at host 192.168.201.152:988: is it running a compatible version of Lustre? [ 7966.257391] LNetError: Skipped 7 previous similar messages [ 7966.961214] LustreError: 463060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783079086 with bad export cookie 12014549984316841169 [ 7966.965044] LustreError: 463060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7966.965300] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7967.085473] Lustre: server umount lustre-MDT0001 complete [ 7970.937489] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7971.849725] Key type lgssc unregistered [ 7971.999421] LNet: 466899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7972.002889] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7972.005280] LNet: Removed LNI 192.168.201.152@tcp [ 7973.023158] LNet: Removed LNI 192.168.201.152@tcp100 [ 7973.025647] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7973.029723] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7973.034121] LNet: Removed LNI 192.168.201.152@tcp101 [ 7974.047245] LNet: Removed LNI 192.168.201.152@tcp102 [ 7974.050308] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7974.055359] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7974.060880] LNet: Removed LNI 192.168.201.152@tcp103 [ 7975.071220] LNet: Removed LNI 192.168.201.152@tcp104 [ 7976.095147] LNet: Removed LNI 192.168.201.152@tcp105 [ 7976.097040] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7976.100092] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7976.103267] LNet: Removed LNI 192.168.201.152@tcp106 [ 7977.119202] LNet: Removed LNI 192.168.201.152@tcp107 [ 7978.143176] LNet: Removed LNI 192.168.201.152@tcp108 [ 7979.167238] LNet: Removed LNI 192.168.201.152@tcp109 [ 7980.191225] LNet: Removed LNI 192.168.201.152@tcp110 [ 7980.193986] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7980.198637] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7980.203864] LNet: Removed LNI 192.168.201.152@tcp111 [ 7981.215189] LNet: Removed LNI 192.168.201.152@tcp112 [ 7982.239210] LNet: Removed LNI 192.168.201.152@tcp113 [ 7983.263289] LNet: Removed LNI 192.168.201.152@tcp114 [ 7984.287212] LNet: Removed LNI 192.168.201.152@tcp115 [ 7985.311151] LNet: Removed LNI 192.168.201.152@tcp116 [ 7986.335282] LNet: Removed LNI 192.168.201.152@tcp117 [ 7987.359190] LNet: Removed LNI 192.168.201.152@tcp118 [ 7988.383200] LNet: Removed LNI 192.168.201.152@tcp119 [ 7988.385966] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7988.390692] LNetError: 466899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7988.395853] LNet: Removed LNI 192.168.201.152@tcp120 [ 7989.407256] LNet: Removed LNI 192.168.201.152@tcp121 [ 7990.431121] LNet: Removed LNI 192.168.201.152@tcp122 [ 7991.455225] LNet: Removed LNI 192.168.201.152@tcp123 [ 7992.479149] LNet: Removed LNI 192.168.201.152@tcp124 [ 7993.503124] LNet: Removed LNI 192.168.201.152@tcp125 [ 7994.527139] LNet: Removed LNI 192.168.201.152@tcp126 [ 7995.551189] LNet: Removed LNI 192.168.201.152@tcp127 [ 7996.575233] LNet: Removed LNI 192.168.201.152@tcp128 [ 7997.599157] LNet: Removed LNI 192.168.201.152@tcp129 [ 7997.601634] LNet: Removed LNI 192.168.201.152@tcp130 [ 7997.604416] LNet: Removed LNI 192.168.201.152@tcp131 [ 7997.607094] LNet: Removed LNI 192.168.201.152@tcp132 [ 7997.609641] LNet: Removed LNI 192.168.201.152@tcp133 [ 7997.612327] LNet: Removed LNI 192.168.201.152@tcp134 [ 7997.620799] LNet: Removed LNI 192.168.201.152@tcp135 [ 7997.971858] Key type .llcrypt unregistered [ 7997.974207] Key type ._llcrypt unregistered [ 8006.758766] Key type ._llcrypt registered [ 8006.759846] Key type .llcrypt registered [ 8006.794590] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8013.968192] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8014.358402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8014.370152] alg: No test for adler32 (adler32-zlib) [ 8015.251527] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 8015.368502] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8016.943232] Key type lgssc registered [ 8017.389962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.158570] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:45:49 (1783079149) [ 8037.762167] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8038.940050] Key type lgssc unregistered [ 8039.095547] LNet: 470957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8039.100069] LNetError: 470957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8039.111530] LNet: Removed LNI 192.168.201.152@tcp [ 8039.531294] Key type .llcrypt unregistered [ 8039.532536] Key type ._llcrypt unregistered [ 8047.010167] Key type ._llcrypt registered [ 8047.011248] Key type .llcrypt registered [ 8047.048176] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8047.307624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8047.319303] alg: No test for adler32 (adler32-zlib) [ 8048.163554] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 8048.250889] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8049.823187] Key type lgssc registered [ 8050.201320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8053.739286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8053.749510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8054.851718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8054.864322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8054.905783] Lustre: lustre-MDT0000: new disk, initializing [ 8054.928602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.934462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8056.491852] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8061.450597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.512565] Lustre: 472145:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8061.552381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8061.557061] Lustre: Skipped 1 previous similar message [ 8061.602103] Lustre: lustre-MDT0001: new disk, initializing [ 8061.638097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8061.651073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8061.654303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8063.090373] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8067.070451] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8069.175917] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8071.953268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.077801] Lustre: lustre-OST0000: new disk, initializing [ 8072.080425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8072.084257] Lustre: 474116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8072.121436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8074.228738] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8078.275565] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8078.343912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8078.347617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8078.385196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8079.611847] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing lnet_if_list [ 8081.340839] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8081.356874] LNet: Added LNI 192.168.201.152@tcp10 [8/256/0/180] [ 8083.589976] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8083.607503] LNet: Added LNI 192.168.201.152@tcp11 [8/256/0/180] [ 8085.352794] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8085.366447] LNet: Added LNI 192.168.201.152@tcp12 [8/256/0/180] [ 8087.086642] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8087.103499] LNet: Added LNI 192.168.201.152@tcp13 [8/256/0/180] [ 8088.792372] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8088.804787] LNet: Added LNI 192.168.201.152@tcp14 [8/256/0/180] [ 8090.418816] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8090.437667] LNet: Added LNI 192.168.201.152@tcp15 [8/256/0/180] [ 8092.215540] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8092.231652] LNet: Added LNI 192.168.201.152@tcp16 [8/256/0/180] [ 8093.961159] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8093.971734] LNet: Added LNI 192.168.201.152@tcp17 [8/256/0/180] [ 8095.779444] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8095.792759] LNet: Added LNI 192.168.201.152@tcp18 [8/256/0/180] [ 8097.727159] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8097.742685] LNet: Added LNI 192.168.201.152@tcp19 [8/256/0/180] [ 8099.780862] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8099.796054] LNet: Added LNI 192.168.201.152@tcp20 [8/256/0/180] [ 8101.581647] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8101.592187] LNet: Added LNI 192.168.201.152@tcp21 [8/256/0/180] [ 8103.513476] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8103.528046] LNet: Added LNI 192.168.201.152@tcp22 [8/256/0/180] [ 8105.509147] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8105.520651] LNet: Added LNI 192.168.201.152@tcp23 [8/256/0/180] [ 8107.466410] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8107.477885] LNet: Added LNI 192.168.201.152@tcp24 [8/256/0/180] [ 8109.672712] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8109.684744] LNet: Added LNI 192.168.201.152@tcp25 [8/256/0/180] [ 8111.832910] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8111.846532] LNet: Added LNI 192.168.201.152@tcp26 [8/256/0/180] [ 8114.000213] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8114.010591] LNet: Added LNI 192.168.201.152@tcp27 [8/256/0/180] [ 8116.163642] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8116.176865] LNet: Added LNI 192.168.201.152@tcp28 [8/256/0/180] [ 8118.274137] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8118.285412] LNet: Added LNI 192.168.201.152@tcp29 [8/256/0/180] [ 8120.607235] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8120.623502] LNet: Added LNI 192.168.201.152@tcp30 [8/256/0/180] [ 8122.853122] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8122.871547] LNet: Added LNI 192.168.201.152@tcp31 [8/256/0/180] [ 8125.212898] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8125.222883] LNet: Added LNI 192.168.201.152@tcp32 [8/256/0/180] [ 8127.085347] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8127.096201] LNet: Added LNI 192.168.201.152@tcp33 [8/256/0/180] [ 8129.012867] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8129.022040] LNet: Added LNI 192.168.201.152@tcp34 [8/256/0/180] [ 8131.093626] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8131.108308] LNet: Added LNI 192.168.201.152@tcp35 [8/256/0/180] [ 8133.415360] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8133.429670] LNet: Added LNI 192.168.201.152@tcp36 [8/256/0/180] [ 8136.177396] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8136.193839] LNet: Added LNI 192.168.201.152@tcp37 [8/256/0/180] [ 8140.693254] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8140.714104] LNet: Added LNI 192.168.201.152@tcp38 [8/256/0/180] [ 8144.966940] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8144.992183] LNet: Added LNI 192.168.201.152@tcp39 [8/256/0/180] [ 8149.117653] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8149.136977] LNet: Added LNI 192.168.201.152@tcp40 [8/256/0/180] [ 8153.767199] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8153.796799] LNet: Added LNI 192.168.201.152@tcp41 [8/256/0/180] [ 8158.079523] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8158.092494] LNet: Added LNI 192.168.201.152@tcp42 [8/256/0/180] [ 8161.074917] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8161.089939] LNet: Added LNI 192.168.201.152@tcp43 [8/256/0/180] [ 8163.436660] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8163.448991] LNet: Added LNI 192.168.201.152@tcp44 [8/256/0/180] [ 8165.783753] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8165.796853] LNet: Added LNI 192.168.201.152@tcp45 [8/256/0/180] [ 8168.298130] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8168.310910] LNet: Added LNI 192.168.201.152@tcp46 [8/256/0/180] [ 8170.885641] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8170.897727] LNet: Added LNI 192.168.201.152@tcp47 [8/256/0/180] [ 8173.414366] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8173.424291] LNet: Added LNI 192.168.201.152@tcp48 [8/256/0/180] [ 8175.994944] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8176.007188] LNet: Added LNI 192.168.201.152@tcp49 [8/256/0/180] [ 8178.852886] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8178.863074] LNet: Added LNI 192.168.201.152@tcp50 [8/256/0/180] [ 8189.919856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8189.923232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.926901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.945124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.945691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.946923] Lustre: Skipped 1 previous similar message [ 8190.950473] Lustre: Skipped 1 previous similar message [ 8192.281604] Lustre: server umount lustre-MDT0000 complete [ 8193.543417] LustreError: 472942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783079312 with bad export cookie 18018192445349095224 [ 8193.546663] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.548078] LustreError: 472942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8193.685825] Lustre: server umount lustre-MDT0001 complete [ 8200.680128] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8203.392736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.447245] Lustre: 471420:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783079315/real 1783079315] req@00000000e075dd14 x1869694020705664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783079331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8212.456169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8214.239153] Lustre: 471421:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783079317/real 1783079317] req@00000000fe19be57 x1869694020705792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783079333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8217.567155] Lustre: 471422:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783079320/real 1783079320] req@00000000fa32ae45 x1869694020705920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783079336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8218.591412] LustreError: 471419:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f70f5d85 x1869694020706816/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8218.705169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.743128] Lustre: 471423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783079322/real 1783079322] req@000000000a27acdc x1869694020706176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783079338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8220.117170] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8223.009188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.143059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8224.553247] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8224.735217] Lustre: 471420:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783079327/real 1783079327] req@0000000089fac006 x1869694020706304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783079343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8226.967192] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8228.324118] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8228.791915] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8231.396063] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8236.513769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.514901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8236.517420] Lustre: Skipped 1 previous similar message [ 8236.520428] Lustre: Skipped 1 previous similar message [ 8241.632523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.634387] Lustre: Skipped 1 previous similar message [ 8242.458048] Lustre: server umount lustre-OST0000 complete [ 8243.679657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8243.682053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.752151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.754831] Lustre: Skipped 2 previous similar messages [ 8249.816854] Lustre: server umount lustre-MDT0000 complete [ 8250.996113] LustreError: 472137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783079370 with bad export cookie 18018192445349097289 [ 8250.998969] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.000229] LustreError: 472137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8251.104450] Lustre: server umount lustre-MDT0001 complete [ 8254.696423] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8255.706345] Key type lgssc unregistered [ 8255.862544] LNet: 501081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8255.866098] LNetError: 501081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8255.868435] LNet: Removed LNI 192.168.201.152@tcp [ 8255.869445] LNet: Removed LNI 192.168.201.152@tcp10 [ 8255.870533] LNet: Removed LNI 192.168.201.152@tcp11 [ 8255.871574] LNet: Removed LNI 192.168.201.152@tcp12 [ 8255.872713] LNet: Removed LNI 192.168.201.152@tcp13 [ 8255.874172] LNet: Removed LNI 192.168.201.152@tcp14 [ 8255.875114] LNet: Removed LNI 192.168.201.152@tcp15 [ 8255.876155] LNet: Removed LNI 192.168.201.152@tcp16 [ 8255.877185] LNet: Removed LNI 192.168.201.152@tcp17 [ 8255.878829] LNet: Removed LNI 192.168.201.152@tcp18 [ 8255.880819] LNet: Removed LNI 192.168.201.152@tcp19 [ 8255.882796] LNet: Removed LNI 192.168.201.152@tcp20 [ 8255.884621] LNet: Removed LNI 192.168.201.152@tcp21 [ 8255.885670] LNet: Removed LNI 192.168.201.152@tcp22 [ 8255.886769] LNet: Removed LNI 192.168.201.152@tcp23 [ 8255.887819] LNet: Removed LNI 192.168.201.152@tcp24 [ 8255.889060] LNet: Removed LNI 192.168.201.152@tcp25 [ 8255.890105] LNet: Removed LNI 192.168.201.152@tcp26 [ 8255.891134] LNet: Removed LNI 192.168.201.152@tcp27 [ 8255.892337] LNet: Removed LNI 192.168.201.152@tcp28 [ 8255.893383] LNet: Removed LNI 192.168.201.152@tcp29 [ 8255.894561] LNet: Removed LNI 192.168.201.152@tcp30 [ 8255.896111] LNet: Removed LNI 192.168.201.152@tcp31 [ 8255.897193] LNet: Removed LNI 192.168.201.152@tcp32 [ 8255.898249] LNet: Removed LNI 192.168.201.152@tcp33 [ 8255.899336] LNet: Removed LNI 192.168.201.152@tcp34 [ 8255.900353] LNet: Removed LNI 192.168.201.152@tcp35 [ 8255.901463] LNet: Removed LNI 192.168.201.152@tcp36 [ 8255.902950] LNet: Removed LNI 192.168.201.152@tcp37 [ 8255.904344] LNet: Removed LNI 192.168.201.152@tcp38 [ 8255.906190] LNet: Removed LNI 192.168.201.152@tcp39 [ 8255.907350] LNet: Removed LNI 192.168.201.152@tcp40 [ 8255.909139] LNet: Removed LNI 192.168.201.152@tcp41 [ 8255.910334] LNet: Removed LNI 192.168.201.152@tcp42 [ 8255.911406] LNet: Removed LNI 192.168.201.152@tcp43 [ 8255.912296] LNet: Removed LNI 192.168.201.152@tcp44 [ 8255.913263] LNet: Removed LNI 192.168.201.152@tcp45 [ 8255.914182] LNet: Removed LNI 192.168.201.152@tcp46 [ 8255.915191] LNet: Removed LNI 192.168.201.152@tcp47 [ 8255.916182] LNet: Removed LNI 192.168.201.152@tcp48 [ 8255.917195] LNet: Removed LNI 192.168.201.152@tcp49 [ 8255.924404] LNet: Removed LNI 192.168.201.152@tcp50 [ 8256.192502] Key type .llcrypt unregistered [ 8256.193615] Key type ._llcrypt unregistered [ 8262.350869] Key type ._llcrypt registered [ 8262.351926] Key type .llcrypt registered [ 8262.390541] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8268.223451] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8268.611763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8268.618684] alg: No test for adler32 (adler32-zlib) [ 8269.467487] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 8269.554896] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8271.135178] Key type lgssc registered [ 8271.483316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8282.625726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8282.631213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.737277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8283.749520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8283.792950] Lustre: lustre-MDT0000: new disk, initializing [ 8283.818079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.825412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8285.266540] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8289.641885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.688736] Lustre: 504249:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8289.719393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8289.721291] Lustre: Skipped 1 previous similar message [ 8289.755909] Lustre: lustre-MDT0001: new disk, initializing [ 8289.797760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8289.816915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8289.820687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8291.090909] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8294.721257] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8296.487108] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8299.056527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8299.131561] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8299.159247] Lustre: lustre-OST0000: new disk, initializing [ 8299.160812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8299.163035] Lustre: 506264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8299.190153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8301.103045] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8305.440273] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8308.228133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8308.232279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8308.267877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8313.313029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8313.314704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8313.316266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8313.318740] Lustre: Skipped 1 previous similar message [ 8314.510056] Lustre: server umount lustre-OST0000 complete [ 8318.432689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.433456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.438952] Lustre: Skipped 1 previous similar message [ 8318.441714] Lustre: Skipped 1 previous similar message [ 8321.889341] Lustre: server umount lustre-MDT0000 complete [ 8323.278907] LustreError: 505044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783079442 with bad export cookie 732947236716984560 [ 8323.283267] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8323.286512] LustreError: 505044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8323.426617] Lustre: server umount lustre-MDT0001 complete [ 8327.273474] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8328.780771] Key type lgssc unregistered [ 8328.947685] LNet: 508170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8328.952585] LNetError: 508170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8328.963550] LNet: Removed LNI 192.168.201.152@tcp [ 8329.414410] Key type .llcrypt unregistered [ 8329.417281] Key type ._llcrypt unregistered [ 8338.561306] Key type ._llcrypt registered [ 8338.562359] Key type .llcrypt registered [ 8338.594714] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8338.898498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8338.924967] alg: No test for adler32 (adler32-zlib) [ 8339.783258] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 8339.884895] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8341.463135] Key type lgssc registered [ 8341.819530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8344.270346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8344.276448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8345.413552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.685433] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8350.174174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8350.326958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.647166] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8354.876667] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8357.542320] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8360.633062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8360.792198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8362.943668] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8366.149395] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8380.987108] loop0: detected capacity change from 0 to 204800000 [ 8381.088912] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8381.095054] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8381.101501] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8383.953715] loop0: detected capacity change from 0 to 204800000 [ 8384.030246] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8384.034566] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8384.040575] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8386.451666] loop0: detected capacity change from 0 to 204800000 [ 8389.078152] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8389.167981] Lustre: fs15246-OST0000: new disk, initializing [ 8389.172647] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8389.179815] Lustre: 513029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8389.215529] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.894610] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8396.937828] loop1: detected capacity change from 0 to 204800000 [ 8399.420263] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8399.477870] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8399.509867] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8399.557118] Lustre: fs15246-MDT0000: new disk, initializing [ 8399.636945] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.652725] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8399.722901] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8400.736351] Lustre: 514154:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000072432aba x1869694325784832/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783079535 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8401.406715] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8405.793370] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8405.798827] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8405.807622] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8415.207132] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.216738] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.585128] Lustre: server umount fs15246-MDT0000 complete [ 8420.327495] LustreError: 509411:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.338438] LustreError: 509411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8421.906893] Lustre: server umount fs15246-OST0000 complete [ 8430.561498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8430.566913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8430.570311] Lustre: Skipped 1 previous similar message [ 8432.736811] Lustre: server umount lustre-OST0000 complete [ 8435.682038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.689775] Lustre: Skipped 1 previous similar message [ 8435.693354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8435.696850] Lustre: Skipped 1 previous similar message [ 8440.557949] Lustre: server umount lustre-MDT0000 complete [ 8440.800848] LustreError: 509415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8440.810676] LustreError: 509415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8442.274854] LustreError: 509397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783079561 with bad export cookie 15499977576099594776 [ 8442.280222] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.282427] LustreError: 509397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8442.444694] Lustre: server umount lustre-MDT0001 complete [ 8446.957862] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8448.368819] Key type lgssc unregistered [ 8448.546616] LNet: 517246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8448.551767] LNetError: 517246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8448.562515] LNet: Removed LNI 192.168.201.152@tcp [ 8449.028862] Key type .llcrypt unregistered [ 8449.031164] Key type ._llcrypt unregistered [ 8458.152634] Key type ._llcrypt registered [ 8458.154283] Key type .llcrypt registered [ 8458.202716] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8458.614627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8458.622680] alg: No test for adler32 (adler32-zlib) [ 8459.487525] Lustre: Lustre: Build Version: 2.17.54_83_g8e46928 [ 8459.598901] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8461.183220] Key type lgssc registered [ 8461.771980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8479.030856] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8482.966518] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8498.299913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8498.312050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.403760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8499.414779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8499.461546] Lustre: lustre-MDT0000: new disk, initializing [ 8499.486225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8499.493984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8501.350529] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8506.847822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.917548] Lustre: 521681:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8506.953767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8506.957251] Lustre: Skipped 1 previous similar message [ 8507.000217] Lustre: lustre-MDT0001: new disk, initializing [ 8507.038259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8507.053802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8507.057994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8508.779608] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8513.643318] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8516.054837] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8519.233566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.374903] Lustre: lustre-OST0000: new disk, initializing [ 8519.377023] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8519.380296] Lustre: 523650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8519.412365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.985829] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8526.772803] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8528.698192] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8528.905901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8528.910653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8528.953611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8529.853838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 8531.543815] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8531.629900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8533.987617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8533.989188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.993199] LustreError: Skipped 1 previous similar message [ 8533.994250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8534.000577] Lustre: Skipped 1 previous similar message [ 8538.677390] Lustre: server umount lustre-OST0000 complete [ 8543.200674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8543.205300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.213753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.217659] Lustre: Skipped 1 previous similar message [ 8544.225906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8544.227624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.741363] Lustre: server umount lustre-MDT0000 complete [ 8548.344810] LustreError: 522477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783079667 with bad export cookie 7132892785017195808 [ 8548.351244] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8548.352384] LustreError: 522477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8548.525656] Lustre: server umount lustre-MDT0001 complete [ 8558.928563] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8562.607379] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8575.316617] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8256 sec ========== 07:54:53 (1783079693) [ 8575.982862] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:54:54 (1783079694) === [ 8577.530150] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:54:55 (1783079695) === [ 8594.559555] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8596.089581] Key type lgssc unregistered [ 8596.255703] LNet: 530290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8596.260633] LNetError: 530290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8596.273843] LNet: Removed LNI 192.168.201.152@tcp [ 8596.749764] Key type .llcrypt unregistered [ 8596.751852] Key type ._llcrypt unregistered