[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 574371563 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003281] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010120] LSM: Security Framework initializing [ 0.011031] Yama: becoming mindful. [ 0.012023] SELinux: Initializing. [ 0.013042] *** VALIDATE selinux *** [ 0.020151] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025160] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] *** VALIDATE tmpfs *** [ 0.030206] *** VALIDATE proc *** [ 0.031129] *** VALIDATE cgroup *** [ 0.031826] *** VALIDATE cgroup2 *** [ 0.032206] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033120] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035024] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039743] debug: unmapping init [mem 0xffffffff85a59000-0xffffffff85a60fff] [ 0.041144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042579] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043094] ... version: 2 [ 0.043894] ... bit width: 48 [ 0.044015] ... generic registers: 4 [ 0.044933] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046011] ... fixed-purpose events: 3 [ 0.046738] ... event mask: 000000070000000f [ 0.047352] rcu: Hierarchical SRCU implementation. [ 0.049167] smp: Bringing up secondary CPUs ... [ 0.050542] x86: Booting SMP configuration: [ 0.051029] .... node #0, CPUs: #1 #2 #3 [ 0.060098] smp: Brought up 1 node, 4 CPUs [ 0.061992] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.090427] node 0 deferred pages initialised in 27ms [ 0.093145] devtmpfs: initialized [ 0.093979] x86/mm: Memory block size: 128MB [ 0.096636] gcov: version magic: 0x41383552 [ 0.098303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.099067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.100312] pinctrl core: initialized pinctrl subsystem [ 0.101153] [ 0.101635] ************************************************************* [ 0.104010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.105008] ** ** [ 0.107014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.108009] ** ** [ 0.110009] ** This means that this kernel is built to expose internal ** [ 0.112029] ** IOMMU data structures, which may compromise security on ** [ 0.114011] ** your system. ** [ 0.115007] ** ** [ 0.117009] ** If you see this message and you are not debugging the ** [ 0.119010] ** kernel, report this immediately to your vendor! ** [ 0.120008] ** ** [ 0.122013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124009] ************************************************************* [ 0.125670] NET: Registered protocol family 16 [ 0.127424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.131051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.134019] cpuidle: using governor menu [ 0.135645] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137464] PCI: Using configuration type 1 for base access [ 0.139104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.146108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.148026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.150121] cryptd: max_cpu_qlen set to 1000 [ 0.152013] ACPI: Added _OSI(Module Device) [ 0.153007] ACPI: Added _OSI(Processor Device) [ 0.154008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.155008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.157000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.164743] ACPI: Interpreter enabled [ 0.167056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.168007] ACPI: Using IOAPIC for interrupt routing [ 0.170098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.174496] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.189861] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.192134] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.195013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.198092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.203202] acpiphp: Slot [2] registered [ 0.205114] acpiphp: Slot [5] registered [ 0.207253] acpiphp: Slot [6] registered [ 0.208238] acpiphp: Slot [7] registered [ 0.210126] acpiphp: Slot [8] registered [ 0.212128] acpiphp: Slot [9] registered [ 0.213233] acpiphp: Slot [10] registered [ 0.215112] acpiphp: Slot [3] registered [ 0.217168] acpiphp: Slot [4] registered [ 0.219097] acpiphp: Slot [11] registered [ 0.220086] acpiphp: Slot [12] registered [ 0.222111] acpiphp: Slot [13] registered [ 0.224103] acpiphp: Slot [14] registered [ 0.225085] acpiphp: Slot [15] registered [ 0.227187] acpiphp: Slot [16] registered [ 0.228074] acpiphp: Slot [17] registered [ 0.230087] acpiphp: Slot [18] registered [ 0.231096] acpiphp: Slot [19] registered [ 0.233160] acpiphp: Slot [20] registered [ 0.234081] acpiphp: Slot [21] registered [ 0.236099] acpiphp: Slot [22] registered [ 0.238165] acpiphp: Slot [23] registered [ 0.239110] acpiphp: Slot [24] registered [ 0.241135] acpiphp: Slot [25] registered [ 0.243115] acpiphp: Slot [26] registered [ 0.244096] acpiphp: Slot [27] registered [ 0.246112] acpiphp: Slot [28] registered [ 0.247092] acpiphp: Slot [29] registered [ 0.249092] acpiphp: Slot [30] registered [ 0.251088] acpiphp: Slot [31] registered [ 0.252052] PCI host bridge to bus 0000:00 [ 0.253019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.261024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.263019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.266032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.269246] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.271467] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275409] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.285857] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.292095] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.295018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.300105] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304419] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.306875] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.310042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.313961] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.320015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.336018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.341014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348244] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.354051] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.362017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.386868] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.393018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.397000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.411030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.419963] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.428015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.433016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.446020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.455206] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.461018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.467020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.481016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.490559] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.495016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.501017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.513015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.520000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.527074] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.537034] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.561022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.570712] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.573344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.575273] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.577218] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.579194] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.584117] iommu: Default domain type: Passthrough [ 0.586961] SCSI subsystem initialized [ 0.589207] ACPI: bus type USB registered [ 0.590064] usbcore: registered new interface driver usbfs [ 0.592062] usbcore: registered new interface driver hub [ 0.593161] usbcore: registered new device driver usb [ 0.595165] pps_core: LinuxPPS API ver. 1 registered [ 0.597007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.599063] PTP clock support registered [ 0.601196] EDAC MC: Ver: 3.0.0 [ 0.602111] PCI: Using ACPI for IRQ routing [ 0.604823] NetLabel: Initializing [ 0.606144] NetLabel: domain hash size = 128 [ 0.608015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.611101] NetLabel: unlabeled traffic allowed by default [ 0.613064] vgaarb: loaded [ 0.617733] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.621015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.629000] clocksource: Switched to clocksource kvm-clock [ 0.912697] VFS: Disk quotas dquot_6.6.0 [ 0.917142] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.921018] *** VALIDATE ramfs *** [ 0.922944] *** VALIDATE hugetlbfs *** [ 0.925360] pnp: PnP ACPI init [ 0.929160] pnp: PnP ACPI: found 6 devices [ 0.957437] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.962095] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.965155] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.968111] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.970762] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.973136] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.977087] NET: Registered protocol family 2 [ 0.980515] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.987566] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.992001] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.998379] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.006749] TCP: Hash tables configured (established 65536 bind 65536) [ 1.010381] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.013953] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018119] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024153] NET: Registered protocol family 1 [ 1.028722] RPC: Registered named UNIX socket transport module. [ 1.031738] RPC: Registered udp transport module. [ 1.034661] RPC: Registered tcp transport module. [ 1.037348] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041282] NET: Registered protocol family 44 [ 1.043510] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.046123] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048735] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.051555] PCI: CLS 0 bytes, default 64 [ 1.053570] Unpacking initramfs... [ 3.453613] debug: unmapping init [mem 0xffff97907cc54000-0xffff97907ffbffff] [ 3.458131] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.460484] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.463497] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 4.240954] Initialise system trusted keyrings [ 4.243087] Key type blacklist registered [ 4.245469] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.260446] zbud: loaded [ 4.273422] *** VALIDATE nfs *** [ 4.278959] *** VALIDATE nfs4 *** [ 4.281643] pstore: using deflate compression [ 4.292972] Platform Keyring initialized [ 4.480684] NET: Registered protocol family 38 [ 4.484297] Key type asymmetric registered [ 4.487240] Asymmetric key parser 'x509' registered [ 4.488855] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.493361] io scheduler mq-deadline registered [ 4.495323] io scheduler kyber registered [ 4.496946] io scheduler bfq registered [ 4.499558] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.517687] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.520814] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.523746] ACPI: Power Button [PWRF] [ 4.533118] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.540299] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.557072] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.564707] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.581636] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.607206] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.636163] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.641681] Non-volatile memory driver v1.3 [ 4.643053] Linux agpgart interface v0.103 [ 4.695548] virtio_blk virtio1: [vda] 144832 512-byte logical blocks (74.2 MB/70.7 MiB) [ 4.699247] vda: detected capacity change from 0 to 74153984 [ 4.714201] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.716844] vdb: detected capacity change from 0 to 1073741824 [ 4.730221] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.732857] vdc: detected capacity change from 0 to 2621440000 [ 4.748435] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.751326] vdd: detected capacity change from 0 to 2621440000 [ 4.765619] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.768591] vde: detected capacity change from 0 to 4294967296 [ 4.785453] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.788432] vdf: detected capacity change from 0 to 4294967296 [ 4.795485] libphy: Fixed MDIO Bus: probed [ 4.800737] usbcore: registered new interface driver usbserial_generic [ 4.803314] usbserial: USB Serial support registered for generic [ 4.805542] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.809726] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.811733] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.813890] mousedev: PS/2 mouse device common for all mice [ 4.816946] rtc_cmos 00:05: RTC can wake from S4 [ 4.820110] rtc_cmos 00:05: registered as rtc0 [ 4.821919] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.824967] intel_pstate: CPU model not supported [ 4.828304] hid: raw HID events driver (C) Jiri Kosina [ 4.830375] usbcore: registered new interface driver usbhid [ 4.832426] usbhid: USB HID core driver [ 4.833844] drop_monitor: Initializing network drop monitor service [ 4.836025] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.836400] Initializing XFRM netlink socket [ 4.836735] NET: Registered protocol family 10 [ 4.837935] Segment Routing with IPv6 [ 4.837967] NET: Registered protocol family 17 [ 4.838257] mpls_gso: MPLS GSO support [ 4.847318] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.858113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.862833] RAS: Correctable Errors collector initialized. [ 4.865315] AVX version of gcm_enc/dec engaged. [ 4.867052] AES CTR mode by8 optimization enabled [ 4.945940] sched_clock: Marking stable (4945854037, 0)->(5968026222, -1022172185) [ 4.950816] registered taskstats version 1 [ 4.952733] Loading compiled-in X.509 certificates [ 4.955375] zswap: loaded using pool lzo/zbud [ 4.980439] Key type big_key registered [ 4.993648] Key type encrypted registered [ 4.995493] ima: No TPM chip found, activating TPM-bypass! [ 5.000918] ima: Allocated hash algorithm: sha1 [ 5.003227] ima: No architecture policies found [ 5.005648] evm: Initialising EVM extended attributes: [ 5.007897] evm: security.selinux [ 5.009055] evm: security.ima [ 5.010331] evm: security.capability [ 5.011679] evm: HMAC attrs: 0x1 [ 5.014373] rtc_cmos 00:05: setting system clock to 2026-06-26 13:26:19 UTC (1782480379) [ 5.021523] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 5.024656] debug: unmapping init [mem 0xffffffff85782000-0xffffffff85a58fff] [ 5.032167] Write protecting the kernel read-only data: 28672k [ 5.037409] debug: unmapping init [mem 0xffffffff83e03000-0xffffffff83ffffff] [ 5.040310] debug: unmapping init [mem 0xffffffff84714000-0xffffffff847fffff] [ 5.088772] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.103230] systemd[1]: Detected virtualization kvm. [ 5.105394] systemd[1]: Detected architecture x86-64. [ 5.107355] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.136875] systemd[1]: No hostname configured. [ 5.139394] systemd[1]: Set hostname to . [ 5.141800] random: systemd: uninitialized urandom read (16 bytes read) [ 5.144540] systemd[1]: Initializing machine ID from random generator. [ 5.642882] random: systemd: uninitialized urandom read (16 bytes read) [ 5.645501] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.651723] random: systemd: uninitialized urandom read (16 bytes read) [ 5.654301] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.659967] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.260495] device-mapper: uevent: version 1.0.3 [ 7.263888] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.016775] virtio_net virtio0 ens2: renamed from eth0 [ 9.023250] random: fast init done [ 9.375115] scsi host0: ata_piix [ 9.465289] scsi host1: ata_piix [ 9.468990] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.474981] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.523332] random: crng init done [ 14.524653] random: 7 urandom warning(s) missed due to ratelimiting [ 14.630498] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.099390] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.957384] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.690914] SELinux: Disabled at runtime. [ 18.836624] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.852877] systemd[1]: Detected virtualization kvm. [ 18.854880] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.112949] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.121916] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.144965] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.153871] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.163564] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.198575] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.218985] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ 20.339360] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 21.338232] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.070246] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.146105] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.815928] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.931343] EDAC sbridge: Ver: 1.1.2 [ 25.629889] Key type dns_resolver registered [ 26.110112] NFS: Registering the id_resolver key type [ 26.112753] Key type id_resolver registered [ 26.115541] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 101.823701] hrtimer: interrupt took 80693836 ns [ 104.684565] libcfs: loading out-of-tree module taints kernel. [ 104.940117] Key type ._llcrypt registered [ 104.946975] Key type .llcrypt registered [ 105.282726] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 133.022442] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 135.109483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.125940] alg: No test for adler32 (adler32-zlib) [ 136.670800] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 138.182698] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 140.125101] Key type lgssc registered [ 142.709340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 161.632944] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.166451] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 211.290661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 211.315637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.630304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 212.674330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.777418] Lustre: lustre-MDT0000: new disk, initializing [ 212.870244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.881585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.695995] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.842173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.979856] Lustre: 6516:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.025703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.034454] Lustre: Skipped 1 previous similar message [ 230.195681] Lustre: lustre-MDT0001: new disk, initializing [ 230.295409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.338889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.349321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.922667] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.782440] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.105955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.355684] Lustre: lustre-OST0000: new disk, initializing [ 249.359467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.365586] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.436261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.158294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.169287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.306666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 255.822761] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.836419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 270.974946] Lustre: lustre-OST0001: new disk, initializing [ 270.978665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 270.982965] Lustre: 9527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.088995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.665602] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.628900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.647466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.705163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 290.329432] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.800974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.231850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.172123] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 310.789165] Lustre: DEBUG MARKER: Client: 2.17.54.95 [ 313.418730] Lustre: DEBUG MARKER: MDS: 2.17.54.95 [ 315.997260] Lustre: DEBUG MARKER: OSS: 2.17.54.95 [ 317.391244] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 26 09:31:30 EDT 2026 [ 333.711812] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 335.219784] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.727761] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.742487] Lustre: Skipped 2 previous similar messages [ 351.745514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.748987] Lustre: Skipped 1 previous similar message [ 356.729670] Lustre: server umount lustre-MDT0000 complete [ 360.312577] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782480734 with bad export cookie 8010556152038694604 [ 360.320182] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.323330] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 361.952905] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.955619] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.974342] Lustre: Skipped 1 previous similar message [ 361.983442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.999639] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 362.030324] Lustre: Skipped 3 previous similar messages [ 365.991280] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.992184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.027623] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 366.056252] Lustre: Skipped 1 previous similar message [ 366.792330] Lustre: server umount lustre-MDT0001 complete [ 370.686727] Lustre: server umount lustre-OST0000 complete [ 374.634350] Lustre: server umount lustre-OST0001 complete [ 382.040545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 392.749249] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 433.989060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.174313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 434.214332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.297974] Lustre: lustre-MDT0000: new disk, initializing [ 434.375498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.406600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.147658] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 450.556680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.665202] Lustre: 15082:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.722647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.726684] Lustre: Skipped 1 previous similar message [ 450.796130] Lustre: lustre-MDT0001: new disk, initializing [ 450.884452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.933441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.954216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.381935] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 465.291934] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 471.368871] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 479.445247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 479.665909] Lustre: lustre-OST0000: new disk, initializing [ 479.668397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 479.672909] Lustre: 17052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.750536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.485256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 481.501308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.667435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 486.692199] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 496.863207] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 502.073912] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.354365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.628863] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.854709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 512.487203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 512.493567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.498940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.526231] Lustre: Skipped 2 previous similar messages [ 517.602566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 517.616866] Lustre: Skipped 2 previous similar messages [ 524.256573] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 524.532367] Lustre: server umount lustre-OST0000 complete [ 532.962352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.962710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.965933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 532.965943] Lustre: Skipped 2 previous similar messages [ 532.969779] LustreError: Skipped 1 previous similar message [ 532.995963] Lustre: Skipped 1 previous similar message [ 534.595458] Lustre: server umount lustre-MDT0000 complete [ 538.087484] LustreError: 15961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.106061] LustreError: 15961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 538.141934] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782480912 with bad export cookie 8010556152038696130 [ 538.173091] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.610645] Lustre: server umount lustre-MDT0001 complete [ 542.971538] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:35:16 (1782480916) [ 545.283992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 547.083897] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:35:20 (1782480920) [ 553.255841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.830246] LustreError: 19463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.952934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.073849] LustreError: 19464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.155460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 567.796026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.555554] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 581.016388] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 586.427941] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 592.989089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.072173] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 605.248747] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 613.862402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 613.865945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.867147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 613.867156] Lustre: Skipped 1 previous similar message [ 613.909331] Lustre: Skipped 1 previous similar message [ 617.715103] Lustre: server umount lustre-OST0000 complete [ 624.096638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.109037] Lustre: Skipped 1 previous similar message [ 636.383145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 636.776243] Lustre: server umount lustre-MDT0000 complete [ 639.461455] LustreError: 20191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.486529] LustreError: 20191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 640.166431] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481014 with bad export cookie 8010556152038697019 [ 640.170722] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.183130] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 640.515475] Lustre: server umount lustre-MDT0001 complete [ 650.588460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 653.502738] Key type lgssc unregistered [ 653.845516] LNet: 22995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.856386] LNetError: 22995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.875759] LNet: Removed LNI 192.168.202.102@tcp [ 654.832167] Key type .llcrypt unregistered [ 654.833945] Key type ._llcrypt unregistered [ 675.882223] Key type ._llcrypt registered [ 675.884177] Key type .llcrypt registered [ 675.964609] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 677.252704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 677.286695] alg: No test for adler32 (adler32-zlib) [ 678.443801] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 678.947128] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 680.679767] Key type lgssc registered [ 681.789720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.145512] loop: module loaded [ 691.268536] loop0: detected capacity change from 0 to 204800000 [ 691.521963] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.532203] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.764711] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.365319] loop0: detected capacity change from 0 to 204800000 [ 697.556799] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.565441] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.579331] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.633496] loop0: detected capacity change from 0 to 204800000 [ 709.522654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 709.549559] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.841734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 710.908507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 710.996714] Lustre: lustre-MDT0000: new disk, initializing [ 711.094126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.114375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 715.701720] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 724.743284] loop1: detected capacity change from 0 to 204800000 [ 730.863492] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.228810] Lustre: lustre-OST07c6: new disk, initializing [ 731.233778] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 731.242412] Lustre: Skipped 1 previous similar message [ 731.253478] Lustre: 26187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 731.382815] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 737.662470] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 738.880488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 738.886801] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 738.915605] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 742.826280] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 742.846177] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 742.862199] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 742.874602] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 742.883150] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 742.891318] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 757.267313] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 757.280495] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 757.289539] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.301668] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 757.311533] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 757.323896] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.392851] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 762.400475] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.416425] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.423843] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.430937] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.436689] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.441508] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.447942] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.455286] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 762.461802] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.467811] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.473719] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.731420] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 766.744447] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 766.755392] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 766.777242] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.786232] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.792796] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.798341] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 766.804353] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.810349] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 766.819905] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.828137] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 766.835977] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.133481] Lustre: 26851:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 775.141213] Lustre: 26851:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 775.149079] Lustre: 26851:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 775.164116] Lustre: 26851:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 775.179108] Lustre: 26851:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.188357] Lustre: 26851:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 775.201813] Lustre: 26851:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 775.210106] Lustre: 26851:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 775.216716] Lustre: 26851:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.229330] Lustre: 26851:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 775.237788] Lustre: 26851:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.247148] Lustre: 26851:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 786.289322] Lustre: 26851:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 786.308847] Lustre: 26851:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 786.318321] Lustre: 26851:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.325819] Lustre: 26851:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.339846] Lustre: 26851:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.348277] Lustre: 26851:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.357800] Lustre: 26851:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 786.368180] Lustre: 26851:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.372792] Lustre: 26851:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 786.377926] Lustre: 26851:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.384148] Lustre: 26851:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.389606] Lustre: 26851:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 836.067294] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.089450] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.189555] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 846.303201] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 846.305681] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 846.439753] Lustre: server umount lustre-OST07c6 complete [ 855.464499] Lustre: server umount lustre-MDT0000 complete [ 870.911322] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 873.527939] Key type lgssc unregistered [ 873.761778] LNet: 28743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.778458] LNetError: 28743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.855413] LNet: Removed LNI 192.168.202.102@tcp [ 875.812472] Key type .llcrypt unregistered [ 875.817091] Key type ._llcrypt unregistered [ 894.374338] Key type ._llcrypt registered [ 894.377386] Key type .llcrypt registered [ 894.466913] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 896.106720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.135825] alg: No test for adler32 (adler32-zlib) [ 897.331730] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 897.569114] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 899.231471] Key type lgssc registered [ 900.399444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 907.551606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 907.569586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.995751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.888371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 919.533979] LustreError: 30017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.546611] LustreError: 30017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 922.496866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.921537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 927.768067] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 935.172189] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 940.183597] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 947.424839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.627750] LustreError: 31837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.809764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.973546] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 954.875389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 959.222720] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.285405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.289315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.313799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.330934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.340947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.361671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.400544] Lustre: server umount lustre-OST0000 complete [ 984.550263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 984.555220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.590469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 984.603647] Lustre: Skipped 1 previous similar message [ 990.688395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.696837] Lustre: Skipped 3 previous similar messages [ 995.809591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.815930] Lustre: Skipped 1 previous similar message [ 997.343683] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 997.529214] Lustre: server umount lustre-MDT0000 complete [ 999.905458] LustreError: 30016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.957249] LustreError: 30016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1000.997200] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481375 with bad export cookie 13126269195079893040 [ 1001.010174] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.016865] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1001.364974] Lustre: server umount lustre-MDT0001 complete [ 1011.482072] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1014.400582] Key type lgssc unregistered [ 1014.631538] LNet: 33754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.638708] LNetError: 33754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.649709] LNet: Removed LNI 192.168.202.102@tcp [ 1015.568112] Key type .llcrypt unregistered [ 1015.574960] Key type ._llcrypt unregistered [ 1033.027591] Key type ._llcrypt registered [ 1033.029670] Key type .llcrypt registered [ 1033.098516] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1033.991349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.218208] alg: No test for adler32 (adler32-zlib) [ 1035.244265] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1035.434450] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1037.087453] Key type lgssc registered [ 1038.318986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.585675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1045.616849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.070385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.038377] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1057.256564] LustreError: 35021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.271414] LustreError: 35021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1059.201863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.520321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.530513] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1069.827773] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1075.264589] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1081.063962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1081.551584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.878298] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1091.803579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1094.932881] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1104.354869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1104.364512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.381598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1107.425595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1107.429308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.444127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1107.461529] Lustre: Skipped 1 previous similar message [ 1110.448980] Lustre: server umount lustre-OST0000 complete [ 1116.129250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1116.137141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.153887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.423603] Lustre: server umount lustre-MDT0000 complete [ 1122.788700] LustreError: 35022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.571436] LustreError: 35003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481499 with bad export cookie 11523039452692693637 [ 1124.582168] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.586883] LustreError: 35003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1125.000411] Lustre: server umount lustre-MDT0001 complete [ 1136.865552] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1139.450278] Key type lgssc unregistered [ 1139.710478] LNet: 38556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.729413] LNetError: 38556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.741792] LNet: Removed LNI 192.168.202.102@tcp [ 1140.561994] Key type .llcrypt unregistered [ 1140.564268] Key type ._llcrypt unregistered [ 1161.362689] Key type ._llcrypt registered [ 1161.365670] Key type .llcrypt registered [ 1161.479574] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1162.446291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.578442] alg: No test for adler32 (adler32-zlib) [ 1163.707915] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1163.882182] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1165.535246] Key type lgssc registered [ 1166.584765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1173.779679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1173.795722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.344649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.577902] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1184.743553] LustreError: 39952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.757385] LustreError: 39952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1187.299692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.522502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1192.018874] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1199.465907] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1205.064936] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1211.679852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1212.052320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.457995] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1220.294143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1224.593281] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1230.824771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.828748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.847329] Lustre: Skipped 2 previous similar messages [ 1230.865950] Lustre: Skipped 2 previous similar messages [ 1231.659128] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1235.941593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.569227] Lustre: server umount lustre-MDT0000 complete [ 1236.738307] LustreError: 39947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.007441] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481615 with bad export cookie 13532997167383588109 [ 1241.009746] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.018239] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1241.056322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.058766] LustreError: 42205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.073197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1241.084114] Lustre: Skipped 2 previous similar messages [ 1246.176790] LustreError: 39947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.178297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1246.206156] LustreError: 39947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1246.214794] Lustre: Skipped 1 previous similar message [ 1251.297432] LustreError: 42205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.320437] LustreError: 42205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1255.391647] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1255.706289] Lustre: server umount lustre-MDT0001 complete [ 1271.865706] Lustre: server umount lustre-OST0000 complete [ 1291.564987] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1293.904589] Key type lgssc unregistered [ 1294.152226] LNet: 43815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.159518] LNetError: 43815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.168799] LNet: Removed LNI 192.168.202.102@tcp [ 1295.025689] Key type .llcrypt unregistered [ 1295.027272] Key type ._llcrypt unregistered [ 1317.077943] Key type ._llcrypt registered [ 1317.084893] Key type .llcrypt registered [ 1317.237146] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1318.324381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.347774] alg: No test for adler32 (adler32-zlib) [ 1319.467904] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1319.668513] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1321.355145] Key type lgssc registered [ 1322.477757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.126147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.137085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.474946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.369917] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1339.876513] LustreError: 45232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.900736] LustreError: 45232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1343.091831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.518850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1347.990141] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1354.057929] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1358.925869] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1365.164088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1365.539727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.846655] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1374.847210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1378.416778] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1385.259710] Lustre: lustre-OST0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1385.442441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1385.451984] Lustre: Skipped 1 previous similar message [ 1390.052526] Lustre: server umount lustre-OST0000 complete [ 1410.016726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1410.022833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.032525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.037620] Lustre: Skipped 2 previous similar messages [ 1411.041629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.042799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.455620] Lustre: server umount lustre-MDT0000 complete [ 1416.160347] LustreError: 45232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.169528] LustreError: 45232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1416.678657] LustreError: 45968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481791 with bad export cookie 3801046673685660971 [ 1416.682032] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1416.686760] LustreError: 45968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1423.064976] Lustre: server umount lustre-MDT0001 complete [ 1431.339299] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1433.519574] Key type lgssc unregistered [ 1433.711644] LNet: 48940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.715898] LNetError: 48940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.789506] LNet: Removed LNI 192.168.202.102@tcp [ 1435.605844] Key type .llcrypt unregistered [ 1435.608121] Key type ._llcrypt unregistered [ 1457.448221] Key type ._llcrypt registered [ 1457.454386] Key type .llcrypt registered [ 1457.549842] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1458.777425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1458.830460] alg: No test for adler32 (adler32-zlib) [ 1459.884468] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1460.162309] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1461.823762] Key type lgssc registered [ 1462.705821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1468.691775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1468.707548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.114296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.906438] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1479.651557] LustreError: 50356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.667449] LustreError: 50356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1481.794898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.042082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.524544] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1491.752802] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1496.905474] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1502.360317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1502.696752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.441456] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1513.303197] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1516.015287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1522.949783] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1525.690306] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1537.174962] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-26 9h51m50s [ 1538.234937] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1540.375943] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1541.435655] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 1541.602081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.610282] Lustre: Skipped 1 previous similar message [ 1542.641190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1542.646581] Lustre: Skipped 3 previous similar messages [ 1545.823945] Lustre: server umount lustre-MDT0000 complete [ 1546.491729] LustreError: 50352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.967233] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1548.769951] LustreError: 51084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.790222] LustreError: 51084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1552.406654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.486906] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1552.628666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.690393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1553.643175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1555.781247] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1558.287696] Lustre: DEBUG MARKER: Wait for df (36555) ... [ 1559.464924] Lustre: DEBUG MARKER: done [ 1563.104147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.104869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1563.109680] LustreError: Skipped 1 previous similar message [ 1563.112925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1563.118404] Lustre: Skipped 2 previous similar messages [ 1563.150025] Lustre: Skipped 1 previous similar message [ 1568.742653] Lustre: server umount lustre-OST0000 complete [ 1574.385623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.389500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.402777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.407226] Lustre: Skipped 1 previous similar message [ 1574.421338] Lustre: Skipped 3 previous similar messages [ 1577.617967] Lustre: server umount lustre-MDT0000 complete [ 1580.379266] LustreError: 50336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481954 with bad export cookie 5993191794363292855 [ 1580.383922] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.386675] LustreError: 50336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1580.597889] Lustre: server umount lustre-MDT0001 complete [ 1588.558945] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1590.563936] Key type lgssc unregistered [ 1590.833208] LNet: 55282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.841682] LNetError: 55282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.910327] LNet: Removed LNI 192.168.202.102@tcp [ 1592.554259] Key type .llcrypt unregistered [ 1592.555925] Key type ._llcrypt unregistered [ 1623.014786] Key type ._llcrypt registered [ 1623.017576] Key type .llcrypt registered [ 1623.087597] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1623.926346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1623.949544] alg: No test for adler32 (adler32-zlib) [ 1624.936683] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1625.121245] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1626.767131] Key type lgssc registered [ 1627.530776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1633.340224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1633.364807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.562848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1634.601306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1634.756668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.763631] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1644.291419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.368478] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1644.398405] Lustre: 57466:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1644.614723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1647.996824] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1654.025671] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.936240] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1662.559908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.676160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1662.837625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.719670] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1670.750987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1673.092189] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1677.699649] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-26 9h54m11s [ 1678.936029] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1684.046126] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1685.970637] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1686.111867] LustreError: 57474:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1706.271331] Lustre: lustre-MDT0000: Client 3e89e405-fa25-4e1b-88aa-fdc06ac62614 (at 192.168.202.2@tcp) reconnecting [ 1726.183559] LustreError: 57474:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1728.024487] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1729.845163] Lustre: DEBUG MARKER: done [ 1737.184362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1737.191013] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.195141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.201583] Lustre: Skipped 1 previous similar message [ 1737.207680] Lustre: Skipped 1 previous similar message [ 1739.634598] Lustre: server umount lustre-OST0000 complete [ 1742.306594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.317154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.325016] Lustre: Skipped 1 previous similar message [ 1743.392532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.399174] Lustre: Skipped 1 previous similar message [ 1748.450692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.455886] Lustre: Skipped 1 previous similar message [ 1754.600311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.606356] Lustre: Skipped 1 previous similar message [ 1756.127142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1756.215914] Lustre: server umount lustre-MDT0000 complete [ 1758.729226] LustreError: 57461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482133 with bad export cookie 3611425993762330754 [ 1758.731159] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.738734] LustreError: 57461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1758.987795] Lustre: server umount lustre-MDT0001 complete [ 1765.247358] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1766.965543] Key type lgssc unregistered [ 1767.141017] LNet: 61792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.147214] LNetError: 61792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.160703] LNet: Removed LNI 192.168.202.102@tcp [ 1767.650979] Key type .llcrypt unregistered [ 1767.653134] Key type ._llcrypt unregistered [ 1797.626425] Key type ._llcrypt registered [ 1797.627975] Key type .llcrypt registered [ 1797.692685] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1798.396478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1798.459082] alg: No test for adler32 (adler32-zlib) [ 1799.450310] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1799.634677] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1801.271224] Key type lgssc registered [ 1802.171615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.719646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.741902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.917439] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1805.933544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1806.070755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.900415] Lustre: Failing over lustre-MDT0000 [ 1807.085396] Lustre: server umount lustre-MDT0000 complete [ 1811.548387] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:56:25 (1782482185) [ 1815.829162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.083585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.418059] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1823.499485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.575389] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1823.599793] Lustre: 64564:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1823.783919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1826.411447] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1832.430631] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.195584] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.612379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.722592] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1840.830680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.153933] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1850.139291] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.434062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1854.364776] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1859.560850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1859.570141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.585699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.585466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.586389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.885889] Lustre: server umount lustre-MDT0000 complete [ 1863.723438] LustreError: 66430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482238 with bad export cookie 4621329769939314857 [ 1863.732753] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1863.737195] LustreError: 66430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1863.920728] Lustre: server umount lustre-MDT0001 complete [ 1866.443503] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1869.690116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1875.709583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.079675] Lustre: 63492:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782482240/real 1782482240] req@0000000057e5a3c1 x1869068028118400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782482256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1882.095388] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.112665] Lustre: Skipped 1 previous similar message [ 1885.215148] Lustre: 63492:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782482243/real 1782482243] req@000000001c1ed5c9 x1869068028118528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782482259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1887.199276] Lustre: 63490:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782482245/real 1782482245] req@00000000668d09d1 x1869068028118656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782482261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1889.312118] LustreError: 63489:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000022980ae1 x1869068028119552/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1889.618490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.699265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1890.655275] Lustre: 63490:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782482248/real 1782482248] req@00000000eb32a501 x1869068028118912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782482264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1893.717906] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1894.687143] Lustre: 63493:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782482253/real 1782482253] req@000000003153a7ac x1869068028119040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782482269 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1896.928555] LustreError: 68363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.937760] LustreError: 68363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1899.836556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.101119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1901.163589] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1903.377684] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1909.497924] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1913.441063] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1920.213360] Lustre: server umount lustre-MDT0000 complete [ 1920.483694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.500384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.516919] LustreError: 66413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.028885] LustreError: 69070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.052124] LustreError: 69070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1922.716817] LustreError: 64558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482297 with bad export cookie 4621329769939317132 [ 1922.720450] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.724270] LustreError: 64558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1922.945287] Lustre: server umount lustre-MDT0001 complete [ 1927.652441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1935.537615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.519172] Lustre: 63492:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782482301/real 1782482301] req@00000000683c019f x1869068028143488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782482317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1943.537597] Lustre: 63492:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1943.543973] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.553025] Lustre: Skipped 2 previous similar messages [ 1947.944464] LustreError: 66412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.025213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.085232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1952.176908] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1953.258155] LustreError: 66412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.283207] LustreError: 66412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1959.230072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.536820] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1962.539987] Lustre: Skipped 1 previous similar message [ 1962.780176] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1968.211213] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1972.489799] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1977.825125] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.826067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.831514] Lustre: Skipped 1 previous similar message [ 1977.841705] Lustre: Skipped 4 previous similar messages [ 1982.412423] Lustre: server umount lustre-MDT0000 complete [ 1982.946374] LustreError: 71015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.956866] LustreError: 71015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1984.905541] LustreError: 67476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482359 with bad export cookie 4621329769939318511 [ 1984.908716] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.926052] LustreError: 67476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1985.108339] Lustre: server umount lustre-MDT0001 complete [ 1989.682350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1992.281496] Lustre: DEBUG MARKER: files compared the same [ 2004.167852] Lustre: server umount lustre-OST0000 complete [ 2016.003483] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2018.003623] Key type lgssc unregistered [ 2018.198672] LNet: 74288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.206139] LNetError: 74288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.219440] LNet: Removed LNI 192.168.202.102@tcp [ 2018.820485] Key type .llcrypt unregistered [ 2018.823416] Key type ._llcrypt unregistered [ 2035.739890] Key type ._llcrypt registered [ 2035.742445] Key type .llcrypt registered [ 2035.816808] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2036.538710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2036.631029] alg: No test for adler32 (adler32-zlib) [ 2037.660993] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2037.861627] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2039.503233] Key type lgssc registered [ 2040.274209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.212468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.247981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.690141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.713233] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2058.169267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.208308] LustreError: 75684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.221513] LustreError: 75684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2058.475328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2061.672748] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2067.207046] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2070.973699] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2075.633577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2075.929808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.437825] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug +malloc all [ 2086.279348] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2093.351582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2103.776732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.779423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2103.787374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2103.804279] LustreError: Skipped 1 previous similar message [ 2107.276609] Lustre: server umount lustre-OST0000 complete [ 2109.922036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2109.930545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.943685] Lustre: Skipped 1 previous similar message [ 2109.954949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.961985] Lustre: Skipped 1 previous similar message [ 2114.021876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.028297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.035452] Lustre: Skipped 1 previous similar message [ 2115.915311] Lustre: server umount lustre-MDT0000 complete [ 2118.419538] LustreError: 75663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482492 with bad export cookie 10323964918344857459 [ 2118.424716] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.431347] LustreError: 75663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2118.625683] Lustre: server umount lustre-MDT0001 complete [ 2125.797213] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2127.624331] Key type lgssc unregistered [ 2127.838611] LNet: 79248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.851318] LNetError: 79248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.872806] LNet: Removed LNI 192.168.202.102@tcp [ 2128.329976] Key type .llcrypt unregistered [ 2128.332761] Key type ._llcrypt unregistered [ 2144.618721] Key type ._llcrypt registered [ 2144.620755] Key type .llcrypt registered [ 2144.700532] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2145.330865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.411750] alg: No test for adler32 (adler32-zlib) [ 2146.347690] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2146.486282] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2148.095926] Key type lgssc registered [ 2148.831470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2153.831428] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2153.874778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2169.375478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2174.495493] LustreError: 80618:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 2201.120104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.388990] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2210.519636] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2215.485872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.563245] Lustre: *** cfs_fail_loc=706, val=0*** [ 2215.799114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.845046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2218.879445] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2222.053536] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.067191] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2224.561548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.813890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.821944] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2232.701907] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.332492] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2240.492669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.493286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.515589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2243.648577] Lustre: server umount lustre-OST0000 complete [ 2250.722247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.727631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.728931] Lustre: Skipped 2 previous similar messages [ 2250.729296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2250.747342] Lustre: Skipped 2 previous similar messages [ 2252.428827] Lustre: server umount lustre-MDT0000 complete [ 2254.886704] LustreError: 80624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482629 with bad export cookie 9536650673290354301 [ 2254.894751] LustreError: 80624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2254.895078] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.110723] Lustre: server umount lustre-MDT0001 complete [ 2261.864248] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2263.620818] Key type lgssc unregistered [ 2263.818341] LNet: 84228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.825799] LNetError: 84228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.844310] LNet: Removed LNI 192.168.202.102@tcp [ 2264.318586] Key type .llcrypt unregistered [ 2264.319986] Key type ._llcrypt unregistered [ 2280.420813] Key type ._llcrypt registered [ 2280.423932] Key type .llcrypt registered [ 2280.516596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2281.139429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.173707] alg: No test for adler32 (adler32-zlib) [ 2282.112946] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2282.284281] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2283.903289] Key type lgssc registered [ 2284.599563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2289.376750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.399218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.016576] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2298.325789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.669276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.559293] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2303.968571] LustreError: 86309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.987858] LustreError: 86309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2306.766328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.981383] LustreError: 86307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.994951] LustreError: 86307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2307.011793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.028504] LustreError: 86312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.051645] LustreError: 86312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2310.883263] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2314.214909] LustreError: 86313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.239844] LustreError: 86313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2316.457359] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.463840] Lustre: MGS: Client f5f733eb-79c6-4d0d-ac20-4a0cf3867b6e (at 0@lo) reconnecting [ 2316.476764] LustreError: 86323:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2316.602315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.662075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2319.820492] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2332.130767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.138460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2332.143632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2336.861829] Lustre: server umount lustre-OST0000 complete [ 2342.369919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.374168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2342.380037] Lustre: Skipped 1 previous similar message [ 2342.391372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2342.400299] Lustre: Skipped 2 previous similar messages [ 2345.209302] Lustre: server umount lustre-MDT0000 complete [ 2347.490345] LustreError: 86312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.501613] LustreError: 86312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2348.020722] Lustre: server umount lustre-MDT0001 complete [ 2350.567253] Lustre: server umount MGS complete [ 2358.256244] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:05:32 (1782482732) [ 2375.590733] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 2381.058682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2400.351942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.933504] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2408.852818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.962112] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2408.984220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2409.025803] Lustre: lustre-MDT0001: new disk, initializing [ 2409.067917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2409.082426] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2409.091218] Lustre: 93471:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2409.097975] LustreError: 93471:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2411.350345] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2417.370972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2417.498836] Lustre: lustre-OST0000: new disk, initializing [ 2417.500920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2417.506091] Lustre: 94241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2417.508683] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2417.550188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.553630] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2423.777268] Lustre: 94243:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000098dce5cc x1869068533536000/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782482814 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2426.268648] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2430.172551] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.177125] Lustre: MGS: Client 6679d2c0-a3ce-49f1-adb2-177613a6a27b (at 0@lo) reconnecting [ 2430.201379] Lustre: 94239:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2430.218441] LustreError: 93457:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2430.223715] LustreError: 93457:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2430.233938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2430.279540] Lustre: lustre-MDT0000: new disk, initializing [ 2430.363538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.383572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2431.456132] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2431.457854] Lustre: 95297:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000698c8786 x1869068533542400/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782482821 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2431.482365] Lustre: 95297:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 2432.994178] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2436.523790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2436.531870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2436.550871] Lustre: Skipped 1 previous similar message [ 2436.573833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2460.644709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.646328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.655681] Lustre: Skipped 1 previous similar message [ 2460.662738] Lustre: Skipped 1 previous similar message [ 2464.794639] Lustre: server umount lustre-OST0000 complete [ 2470.884684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.888706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2470.895238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.182996] Lustre: server umount lustre-MDT0000 complete [ 2481.907487] Lustre: server umount lustre-MDT0001 complete [ 2484.098728] Lustre: server umount MGS complete [ 2490.773571] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:07:44 (1782482864) [ 2499.820135] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2501.405640] Key type lgssc unregistered [ 2501.588949] LNet: 97809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.593740] LNetError: 97809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.602861] LNet: Removed LNI 192.168.202.102@tcp [ 2502.212106] Key type .llcrypt unregistered [ 2502.213329] Key type ._llcrypt unregistered [ 2512.969261] Key type ._llcrypt registered [ 2512.971770] Key type .llcrypt registered [ 2513.056718] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2513.796774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.814114] alg: No test for adler32 (adler32-zlib) [ 2514.808540] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2514.957341] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2516.583259] Key type lgssc registered [ 2517.492669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2522.997896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.016627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.337515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.846571] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2533.163352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.438517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.281843] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2540.234090] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2543.030466] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2547.761828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.998763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.048966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2552.090278] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2557.141674] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2563.554073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.554825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.560735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.568658] Lustre: Skipped 1 previous similar message [ 2564.865224] Lustre: server umount lustre-OST0000 complete [ 2568.677565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.683742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.694303] Lustre: Skipped 1 previous similar message [ 2568.705571] Lustre: Skipped 2 previous similar messages [ 2573.419406] Lustre: server umount lustre-MDT0000 complete [ 2573.792791] LustreError: 99009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.805892] LustreError: 99009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2575.603657] LustreError: 98990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482950 with bad export cookie 4473946464629716528 [ 2575.606900] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.613378] LustreError: 98990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2575.760972] Lustre: server umount lustre-MDT0001 complete [ 2582.263119] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2583.986872] Key type lgssc unregistered [ 2584.180379] LNet: 102536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2584.186718] LNetError: 102536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.261810] LNet: Removed LNI 192.168.202.102@tcp [ 2585.918962] Key type .llcrypt unregistered [ 2585.921496] Key type ._llcrypt unregistered [ 2595.996263] Key type ._llcrypt registered [ 2595.998192] Key type .llcrypt registered [ 2596.072670] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2596.658986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2596.681968] alg: No test for adler32 (adler32-zlib) [ 2597.603790] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2597.750112] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2599.367259] Key type lgssc registered [ 2600.253612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.316109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2604.326870] LustreError: 103600:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2604.354463] LustreError: 103599:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2604.362458] LustreError: 103600:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 2604.370856] LustreError: 103600:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2604.379400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.380869] LustreError: 103600:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2605.703987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.055024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.227210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2612.568809] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2615.650885] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing lsmod [ 2617.381317] LustreError: 104800:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2617.395096] LustreError: 104806:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2617.403251] LustreError: 104800:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 2617.408183] LustreError: 104800:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2617.414842] LustreError: 104800:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2617.422311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.688528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.384427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.387769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.394082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.409973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.412419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.427106] Lustre: Skipped 1 previous similar message [ 2621.436949] Lustre: Skipped 1 previous similar message [ 2625.709254] Lustre: server umount lustre-MDT0000 complete [ 2626.530520] LustreError: 103632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.540117] LustreError: 103632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2627.992735] LustreError: 103611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483002 with bad export cookie 12519429989629795620 [ 2628.001466] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.663259] LustreError: 104069:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009dcbf011 x1869068864866176/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2629.786230] Lustre: server umount lustre-MDT0001 complete [ 2642.128627] Lustre: server umount lustre-OST0000 complete [ 2647.938690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.205807] LustreError: 105870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.252911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.998074] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2653.671990] LustreError: 105871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.482133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.689706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2659.258507] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2663.488936] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2666.788224] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2670.766829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.974033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.014514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2674.294339] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2678.511670] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2687.458472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.462277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2687.465296] Lustre: Skipped 1 previous similar message [ 2687.472567] Lustre: Skipped 1 previous similar message [ 2687.475830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.579967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.588492] Lustre: Skipped 1 previous similar message [ 2693.403744] Lustre: server umount lustre-OST0000 complete [ 2697.701914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2697.702166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.714858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.715644] Lustre: Skipped 1 previous similar message [ 2697.730831] Lustre: Skipped 1 previous similar message [ 2701.678947] Lustre: server umount lustre-MDT0000 complete [ 2702.818442] LustreError: 105871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.833165] LustreError: 105871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2703.915416] LustreError: 105851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483078 with bad export cookie 12519429989629796306 [ 2703.922502] LustreError: 105851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2703.924770] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.088780] Lustre: server umount lustre-MDT0001 complete [ 2709.763562] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2711.228845] Key type lgssc unregistered [ 2711.454321] LNet: 109401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2711.465505] LNetError: 109401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2711.478524] LNet: Removed LNI 192.168.202.102@tcp [ 2711.928836] Key type .llcrypt unregistered [ 2711.930104] Key type ._llcrypt unregistered [ 2725.037610] Key type ._llcrypt registered [ 2725.039190] Key type .llcrypt registered [ 2725.095345] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2725.504277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2725.525240] alg: No test for adler32 (adler32-zlib) [ 2726.441382] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2726.603337] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2728.215328] Key type lgssc registered [ 2729.016613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2733.439223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2733.455047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.719683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.998381] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2742.064495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.306731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.898186] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2749.142970] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2752.255657] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2757.125342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.360306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.435689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2761.349924] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2765.261373] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2770.093203] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2771.365300] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2776.032681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.037938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.046411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.057043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2777.057800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.061709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.070357] Lustre: Skipped 1 previous similar message [ 2779.446679] Lustre: server umount lustre-OST0000 complete [ 2782.176743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.184266] Lustre: Skipped 1 previous similar message [ 2782.184549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.188198] Lustre: Skipped 1 previous similar message [ 2787.296176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2787.302165] Lustre: Skipped 1 previous similar message [ 2787.558405] Lustre: server umount lustre-MDT0000 complete [ 2789.412520] LustreError: 110775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483163 with bad export cookie 11042310208114200953 [ 2789.418825] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.420789] LustreError: 110775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2789.608081] Lustre: server umount lustre-MDT0001 complete [ 2794.207189] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2795.541413] Key type lgssc unregistered [ 2795.698608] LNet: 114430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.702324] LNetError: 114430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2796.778690] LNet: Removed LNI 192.168.202.102@tcp [ 2797.238215] Key type .llcrypt unregistered [ 2797.239489] Key type ._llcrypt unregistered [ 2806.376263] Key type ._llcrypt registered [ 2806.377685] Key type .llcrypt registered [ 2806.425668] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2806.955952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2806.976876] alg: No test for adler32 (adler32-zlib) [ 2807.858880] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2807.985691] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2809.599249] Key type lgssc registered [ 2810.219320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.352141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2814.361380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.609556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.664100] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2821.932043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.111338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2823.950869] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2827.308355] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2830.253059] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2833.572371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.805500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.416772] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2839.548204] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.002487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2856.418803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2856.419206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.424191] LustreError: Skipped 1 previous similar message [ 2856.435752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.447432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2857.456454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.459633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.265343] Lustre: server umount lustre-OST0000 complete [ 2862.560605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.568405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.572995] Lustre: Skipped 2 previous similar messages [ 2867.640799] Lustre: server umount lustre-MDT0000 complete [ 2867.685322] LustreError: 115630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.698662] LustreError: 115630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2869.545986] LustreError: 115611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483244 with bad export cookie 3992137674454525049 [ 2869.549424] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2869.552637] LustreError: 115611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2869.708158] Lustre: server umount lustre-MDT0001 complete [ 2875.119086] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2876.693850] Key type lgssc unregistered [ 2876.884152] LNet: 119179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2876.888761] LNetError: 119179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2877.926127] LNet: Removed LNI 192.168.202.102@tcp [ 2878.446661] Key type .llcrypt unregistered [ 2878.449744] Key type ._llcrypt unregistered [ 2892.105553] Key type ._llcrypt registered [ 2892.107496] Key type .llcrypt registered [ 2892.184842] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2892.856555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2892.869939] alg: No test for adler32 (adler32-zlib) [ 2893.817544] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2893.982235] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2895.607396] Key type lgssc registered [ 2896.261330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2900.733833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2900.747431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.022870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.086952] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2908.367092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.570216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2910.625347] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2914.804241] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2917.771654] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.294525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.474563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.021215] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2926.637264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2927.655853] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2936.541882] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2937.941802] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2938.580253] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2938.583354] Lustre: Skipped 2 previous similar messages [ 2940.930742] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2940.933433] Lustre: Skipped 2 previous similar messages [ 2946.057599] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2946.071604] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2946.074228] Lustre: Skipped 1 previous similar message [ 2949.969501] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2949.977362] Lustre: Skipped 2 previous similar messages [ 2955.274161] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.2@tcp 192.168.0.[2,10]@tcp [ 2957.876155] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.2/24@tcp in log lustre-MDT0000 [ 2957.880421] Lustre: Skipped 2 previous similar messages [ 2963.489977] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.2/24@tcp [ 2963.494562] Lustre: Skipped 1 previous similar message [ 2967.520094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2967.527517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.534156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2968.548204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2968.548335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2968.549455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.561776] Lustre: Skipped 1 previous similar message [ 2973.619858] Lustre: server umount lustre-OST0000 complete [ 2978.785548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.788903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.795796] Lustre: Skipped 1 previous similar message [ 2978.805052] Lustre: Skipped 1 previous similar message [ 2981.729398] Lustre: server umount lustre-MDT0000 complete [ 2983.633378] LustreError: 124009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483358 with bad export cookie 389543282724750787 [ 2983.637790] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.639484] LustreError: 124009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2983.807770] Lustre: server umount lustre-MDT0001 complete [ 2988.970427] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2990.308543] Key type lgssc unregistered [ 2990.458394] LNet: 125708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2990.461306] LNetError: 125708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2990.476345] LNet: Removed LNI 192.168.202.102@tcp [ 2990.858517] Key type .llcrypt unregistered [ 2990.860497] Key type ._llcrypt unregistered [ 3005.550486] Key type ._llcrypt registered [ 3005.551994] Key type .llcrypt registered [ 3005.614682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3006.096214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3006.140881] alg: No test for adler32 (adler32-zlib) [ 3007.050991] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 3007.165635] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3008.767214] Key type lgssc registered [ 3009.385569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3013.485290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3013.494648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.767747] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3014.773245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.782274] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.2/24@tcp [ 3016.754187] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3021.374762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.553499] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3021.568222] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.2/24@tcp [ 3021.597658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.394395] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3028.494622] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3031.600708] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3035.100558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.300521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.123809] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3040.433848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3041.567708] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3052.003118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.006390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3052.015532] Lustre: Skipped 1 previous similar message [ 3052.021752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.061876] Lustre: server umount lustre-OST0000 complete [ 3057.123499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.128515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.137625] Lustre: Skipped 1 previous similar message [ 3057.143067] Lustre: Skipped 1 previous similar message [ 3062.242529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.244318] Lustre: Skipped 2 previous similar messages [ 3062.318124] Lustre: server umount lustre-MDT0000 complete [ 3063.928322] LustreError: 127225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483438 with bad export cookie 17554511837305041934 [ 3063.931904] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.934617] LustreError: 127225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3064.070474] Lustre: server umount lustre-MDT0001 complete [ 3068.732888] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3070.034825] Key type lgssc unregistered [ 3070.186749] LNet: 130827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3070.189868] LNetError: 130827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3070.198395] LNet: Removed LNI 192.168.202.102@tcp [ 3070.609242] Key type .llcrypt unregistered [ 3070.611358] Key type ._llcrypt unregistered [ 3082.959702] Key type ._llcrypt registered [ 3082.960993] Key type .llcrypt registered [ 3083.012290] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3090.641371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3091.033408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3091.091878] alg: No test for adler32 (adler32-zlib) [ 3092.030431] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 3092.156423] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3093.759343] Key type lgssc registered [ 3094.308826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3110.411283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3110.452401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.581508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3111.596596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3111.635648] Lustre: lustre-MDT0000: new disk, initializing [ 3111.671843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3111.685937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3113.451887] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3118.650028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.701184] Lustre: 134199:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3118.731273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3118.736636] Lustre: Skipped 1 previous similar message [ 3118.776896] Lustre: lustre-MDT0001: new disk, initializing [ 3118.831473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.850415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3118.855176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3120.588155] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3125.173821] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3127.958386] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3130.960349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.101500] Lustre: lustre-OST0000: new disk, initializing [ 3131.104357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3131.109264] Lustre: 136175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3131.167735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.735657] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3138.061628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3138.065653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3138.119112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3138.541349] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3140.785135] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3140.884332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3142.891179] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3143.014668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3148.262325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.264640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.266902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.271403] Lustre: Skipped 1 previous similar message [ 3148.284047] LustreError: Skipped 1 previous similar message [ 3149.995639] Lustre: server umount lustre-OST0000 complete [ 3153.376668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.379120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.382547] Lustre: Skipped 1 previous similar message [ 3153.388503] Lustre: Skipped 2 previous similar messages [ 3157.786935] Lustre: server umount lustre-MDT0000 complete [ 3159.346746] LustreError: 134191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483533 with bad export cookie 11119230275982088701 [ 3159.348463] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.351385] LustreError: 134191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3159.489020] Lustre: server umount lustre-MDT0001 complete [ 3163.366153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.614926] LustreError: 138392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.624562] LustreError: 138392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3163.649446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.388777] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3168.736821] LustreError: 138393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.654691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.869856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3172.047910] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3175.827185] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3178.522666] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3181.974842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.152467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.915625] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3188.355274] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3190.176298] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3190.268901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3191.944053] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3196.201776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3201.993931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.113559] Lustre: lustre-OST0001: Not available for connect from 192.168.202.2@tcp (not set up) [ 3202.119406] Lustre: lustre-OST0001: new disk, initializing [ 3202.122723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3202.127769] Lustre: 141787:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3202.173371] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3204.819469] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3208.534914] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3209.220599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3209.225239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3209.270856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3209.687240] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3211.502468] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3211.607836] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3214.062648] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d8b50f6a800.ost_server_uuid 50 [ 3214.815484] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d8b50f6a800.ost_server_uuid in FULL state after 0 sec [ 3219.434430] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.435175] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.438403] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3219.444054] Lustre: Skipped 1 previous similar message [ 3224.137895] Lustre: server umount lustre-OST0001 complete [ 3224.554026] LustreError: 140211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.561629] LustreError: 140211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3226.596544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3226.601859] LustreError: Skipped 1 previous similar message [ 3226.607041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.627611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3226.630448] Lustre: Skipped 1 previous similar message [ 3229.667307] LustreError: 140212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.674306] LustreError: 140212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3232.153455] Lustre: server umount lustre-OST0000 complete [ 3234.784949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.796584] Lustre: Skipped 1 previous similar message [ 3234.802406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3234.806996] Lustre: Skipped 3 previous similar messages [ 3240.116205] Lustre: server umount lustre-MDT0000 complete [ 3241.844214] LustreError: 138372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483616 with bad export cookie 11119230275982089569 [ 3241.846121] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.854509] LustreError: 138372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3242.012520] Lustre: server umount lustre-MDT0001 complete [ 3251.732415] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3253.097596] Key type lgssc unregistered [ 3253.245479] LNet: 144792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3253.249194] LNetError: 144792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3253.256606] LNet: Removed LNI 192.168.202.102@tcp [ 3253.670682] Key type .llcrypt unregistered [ 3253.672621] Key type ._llcrypt unregistered [ 3273.636284] Key type ._llcrypt registered [ 3273.638053] Key type .llcrypt registered [ 3273.692596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3281.139452] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3281.614831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3281.633798] alg: No test for adler32 (adler32-zlib) [ 3282.518303] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 3282.614994] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3284.207614] Key type lgssc registered [ 3284.746949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3300.250461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3300.262036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.370580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3301.386095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3301.428635] Lustre: lustre-MDT0000: new disk, initializing [ 3301.461610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.468941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3303.059562] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3308.130817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.185842] Lustre: 148921:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3308.218963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3308.221382] Lustre: Skipped 1 previous similar message [ 3308.258594] Lustre: lustre-MDT0001: new disk, initializing [ 3308.302669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3308.328740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3308.334504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3309.886291] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3314.317935] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3316.537842] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3319.404548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.532172] Lustre: lustre-OST0000: new disk, initializing [ 3319.533992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3319.537696] Lustre: 150888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3319.570718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.832158] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3325.957702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3325.961777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3326.005928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3326.099664] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3328.345116] Lustre: Failing over lustre-OST0000 [ 3328.419275] Lustre: server umount lustre-OST0000 complete [ 3341.774361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.908786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.919557] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3342.790617] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3344.025865] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3345.603240] Lustre: Failing over lustre-MDT0000 [ 3345.707568] Lustre: server umount lustre-MDT0000 complete [ 3346.655176] Lustre: 147148:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782483705/real 1782483705] req@00000000ef5f7854 x1869069583150208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782483721 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3346.655294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.664678] Lustre: 147148:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3351.775219] Lustre: 147148:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782483710/real 1782483710] req@000000001e63e05f x1869069583151360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782483726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3352.032780] LustreError: 148931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.039626] LustreError: 148931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3353.333387] LustreError: 148928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.151707] Lustre: 147145:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782483715/real 1782483715] req@0000000081186d3b x1869069583152512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782483731 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3357.153088] LustreError: 150084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.169310] LustreError: 150084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3359.353180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.403559] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3359.407911] Lustre: 148923:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3359.496346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3359.517391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.537631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3361.219827] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3363.574123] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3364.842387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3364.853334] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3364.858141] Lustre: Skipped 1 previous similar message [ 3364.876160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3369.955900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.957314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.962071] LustreError: Skipped 1 previous similar message [ 3369.967030] Lustre: Skipped 3 previous similar messages [ 3369.969756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3369.973078] Lustre: Skipped 1 previous similar message [ 3372.239502] Lustre: server umount lustre-OST0000 complete [ 3375.073503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3375.073985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.075549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.085090] Lustre: Skipped 1 previous similar message [ 3379.805968] Lustre: server umount lustre-MDT0000 complete [ 3380.192222] LustreError: 148931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.198360] LustreError: 148931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3381.221147] LustreError: 148911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483755 with bad export cookie 10942701741590914144 [ 3381.224214] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.226214] LustreError: 148911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3381.359129] Lustre: server umount lustre-MDT0001 complete [ 3385.198438] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3386.331987] Key type lgssc unregistered [ 3386.475402] LNet: 154536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3386.478412] LNetError: 154536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3386.486387] LNet: Removed LNI 192.168.202.102@tcp [ 3386.834845] Key type .llcrypt unregistered [ 3386.836445] Key type ._llcrypt unregistered [ 3408.394853] Key type ._llcrypt registered [ 3408.396323] Key type .llcrypt registered [ 3408.443454] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3408.935171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3408.950420] alg: No test for adler32 (adler32-zlib) [ 3409.849572] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 3409.946980] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3411.535592] Key type lgssc registered [ 3412.027373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3415.207070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3415.213662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.391965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.279685] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3423.081960] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3425.422321] LustreError: 156082:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3425.427985] LustreError: 156082:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3425.431075] LustreError: 156082:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3427.601415] Lustre: server umount lustre-MDT0000 complete [ 3433.496669] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3437.352983] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3451.139064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.226573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3451.239712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3451.283979] Lustre: lustre-MDT0000: new disk, initializing [ 3451.313068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.320630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3452.923376] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3458.025382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.074985] Lustre: 159380:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3458.100920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3458.103043] Lustre: Skipped 1 previous similar message [ 3458.139127] Lustre: lustre-MDT0001: new disk, initializing [ 3458.184527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.207462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3458.214349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3459.729423] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3464.105843] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3466.135857] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3468.805662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.919271] Lustre: lustre-OST0000: new disk, initializing [ 3468.921429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3468.924160] Lustre: 161347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3468.957417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.180874] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3475.458607] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3478.524682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3478.528759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3478.577913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3482.569666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3483.271179] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:24:17 (1782483857) [ 3580.895466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.896134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.902315] Lustre: Skipped 1 previous similar message [ 3586.016130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.019651] Lustre: Skipped 4 previous similar messages [ 3586.854417] Lustre: server umount lustre-MDT0000 complete [ 3588.264096] LustreError: 159375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483962 with bad export cookie 14651871837407519842 [ 3588.264519] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3588.268881] LustreError: 159375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3591.135543] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.136099] LustreError: 161355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.139925] Lustre: Skipped 1 previous similar message [ 3591.146426] LustreError: 161355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3591.147889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3593.183378] LustreError: 163110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.183378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3594.408578] Lustre: server umount lustre-MDT0001 complete [ 3610.079092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3610.122276] Lustre: server umount lustre-OST0000 complete [ 3613.548233] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3617.023781] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3630.048531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.131152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3630.142916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3630.181044] Lustre: lustre-MDT0000: new disk, initializing [ 3630.220336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.234788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3631.767908] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3636.114781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.153940] Lustre: 165977:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3636.178191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3636.180463] Lustre: Skipped 1 previous similar message [ 3636.220410] Lustre: lustre-MDT0001: new disk, initializing [ 3636.265023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3636.268818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3637.655346] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3641.490406] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3643.391586] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3645.685756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.783462] Lustre: lustre-OST0000: new disk, initializing [ 3645.784984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3645.787491] Lustre: 167945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3645.813672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.816466] Lustre: Skipped 1 previous similar message [ 3647.546883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3647.551026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3647.595119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3647.918845] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3651.986789] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3653.563262] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3653.647699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3655.148346] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3655.228388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3657.696026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3657.699502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.704915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.212436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.034623] Lustre: server umount lustre-OST0000 complete [ 3663.328429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.333788] Lustre: Skipped 2 previous similar messages [ 3668.449052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.452782] Lustre: Skipped 5 previous similar messages [ 3669.474282] Lustre: server umount lustre-MDT0000 complete [ 3670.865085] LustreError: 165970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484045 with bad export cookie 14651871837407678658 [ 3670.868375] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.870502] LustreError: 165970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3670.990782] Lustre: server umount lustre-MDT0001 complete [ 3674.043350] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:27:28 (1782484048) [ 3679.592198] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3683.013368] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3695.509198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.595682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3695.607644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3695.645719] Lustre: lustre-MDT0000: new disk, initializing [ 3695.666111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.673094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3697.075670] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3701.434443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.482296] Lustre: 172693:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3701.594484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3703.033400] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3706.902903] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3708.881796] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3711.372624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.496370] Lustre: 174669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3713.681248] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3717.720131] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3719.160837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3719.188191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3729.376973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3729.377611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.379498] LustreError: Skipped 1 previous similar message [ 3729.383234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3729.385666] Lustre: Skipped 1 previous similar message [ 3739.103145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3739.170865] Lustre: server umount lustre-OST0000 complete [ 3742.687886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3742.690897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.695825] Lustre: Skipped 1 previous similar message [ 3746.907055] Lustre: server umount lustre-MDT0000 complete [ 3748.277148] LustreError: 172687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484122 with bad export cookie 14651871837407679526 [ 3748.281085] LustreError: 172687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3748.281363] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.400925] Lustre: server umount lustre-MDT0001 complete [ 3751.529626] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:28:45 (1782484125) [ 3757.314873] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3760.638792] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3774.463854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.613478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3774.638167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3774.640533] Lustre: Skipped 3 previous similar messages [ 3774.678881] Lustre: lustre-MDT0000: new disk, initializing [ 3774.680684] Lustre: Skipped 2 previous similar messages [ 3774.726657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.732084] Lustre: Skipped 2 previous similar messages [ 3774.743328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3774.747139] Lustre: Skipped 2 previous similar messages [ 3776.581196] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3783.407022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.484296] Lustre: 178988:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3783.762307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3787.293922] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3796.050685] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3800.902441] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3806.926152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3807.144491] Lustre: 180963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3812.377958] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3816.006248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3816.123019] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3823.078810] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.901956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.906789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.912196] LustreError: Skipped 1 previous similar message [ 3836.914354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.914363] Lustre: Skipped 5 previous similar messages [ 3836.950663] Lustre: Skipped 2 previous similar messages [ 3839.155802] Lustre: server umount lustre-OST0000 complete [ 3845.090926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3850.139192] Lustre: server umount lustre-MDT0000 complete [ 3852.256983] LustreError: 178994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.273319] LustreError: 178994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3853.368826] LustreError: 178979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484227 with bad export cookie 14651871837407680611 [ 3853.388306] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3853.388529] LustreError: 178979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3853.650075] Lustre: server umount lustre-MDT0001 complete [ 3861.592297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3863.517864] Key type lgssc unregistered [ 3863.745887] LNet: 182819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3863.757252] LNetError: 182819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3863.772485] LNet: Removed LNI 192.168.202.102@tcp [ 3864.438686] Key type .llcrypt unregistered [ 3864.441576] Key type ._llcrypt unregistered [ 3879.745578] Key type ._llcrypt registered [ 3879.747610] Key type .llcrypt registered [ 3879.796714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3880.300549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3880.430581] alg: No test for adler32 (adler32-zlib) [ 3881.399617] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 3881.584893] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 3883.200341] Key type lgssc registered [ 3883.992621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3888.648369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3888.661311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.034071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.052988] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3899.203633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.437914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3902.646452] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3908.042495] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3912.320420] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3917.048957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.260823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.631067] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3926.940357] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3934.712331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3955.168661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.174727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.176093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.191920] Lustre: Skipped 1 previous similar message [ 3958.876493] Lustre: server umount lustre-OST0000 complete [ 3965.415308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.417098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3965.432700] Lustre: Skipped 1 previous similar message [ 3965.441485] Lustre: Skipped 2 previous similar messages [ 3968.165366] Lustre: server umount lustre-MDT0000 complete [ 3970.532238] LustreError: 184213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.543629] LustreError: 184213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3970.898553] LustreError: 184193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484345 with bad export cookie 14548474619700888066 [ 3970.903043] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.909279] LustreError: 184193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3971.161404] Lustre: server umount lustre-MDT0001 complete [ 3983.649515] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 3987.849844] Key type lgssc unregistered [ 3988.748716] LNet: 187767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3988.762635] LNetError: 187767:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3988.800966] LNet: Removed LNI 192.168.202.102@tcp [ 3990.507755] Key type .llcrypt unregistered [ 3990.509560] Key type ._llcrypt unregistered [ 4038.288170] Key type ._llcrypt registered [ 4038.289983] Key type .llcrypt registered [ 4038.376382] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4039.379676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4039.600365] alg: No test for adler32 (adler32-zlib) [ 4040.722706] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 4041.199878] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 4043.149865] Key type lgssc registered [ 4045.733878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4056.675943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4056.702927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.206963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.086584] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4068.325764] LustreError: 189190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.356948] LustreError: 189190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4071.466153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4071.741452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.518786] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4082.184163] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4086.832276] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4092.839257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4093.132641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.353497] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4103.423716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4106.007221] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4112.868071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.890646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.909665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.894044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.902166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.930492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4119.012978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4119.021477] Lustre: Skipped 3 previous similar messages [ 4124.131201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4124.139104] Lustre: Skipped 1 previous similar message [ 4127.203078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4127.386559] Lustre: server umount lustre-OST0000 complete [ 4134.796454] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d8b50f73800.ost_server_uuid 50 [ 4136.699434] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4153.633737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.637814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.663086] Lustre: Skipped 1 previous similar message [ 4153.672676] Lustre: Skipped 1 previous similar message [ 4158.017431] Lustre: server umount lustre-MDT0000 complete [ 4158.956539] LustreError: 189190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.985789] LustreError: 189190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4161.970101] LustreError: 189171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484536 with bad export cookie 11097586716329765735 [ 4161.988194] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.296474] Lustre: server umount lustre-MDT0001 complete [ 4170.491255] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:35:43 (1782484543) [ 4177.157118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.682714] LustreError: 193105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.765481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.017313] LustreError: 193106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.493705] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4188.131209] LustreError: 193105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.041317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4192.458308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.646695] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4204.356644] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4210.106708] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4217.337528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4217.631548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.771931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4224.183485] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4231.766374] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4238.652777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.865557] Lustre: lustre-OST0001: new disk, initializing [ 4238.869050] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4238.873965] Lustre: 195945:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4238.967485] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.714642] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4248.714387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4248.732437] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4248.883450] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4254.317483] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4257.764371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4257.774723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.813450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4257.832528] Lustre: Skipped 1 previous similar message [ 4263.417171] Lustre: server umount lustre-OST0000 complete [ 4264.418271] LustreError: 194927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.445099] LustreError: 194927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4269.467682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4269.676853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4274.133679] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4274.373288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4280.568531] LustreError: 196636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.608966] LustreError: 196636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4290.022866] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4290.035508] LustreError: Skipped 1 previous similar message [ 4290.041494] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.059027] Lustre: Skipped 1 previous similar message [ 4290.066655] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4290.072940] Lustre: Skipped 5 previous similar messages [ 4294.097267] Lustre: server umount lustre-OST0001 complete [ 4300.256538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.269808] Lustre: Skipped 2 previous similar messages [ 4304.286240] Lustre: server umount lustre-MDT0000 complete [ 4308.054049] LustreError: 193087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484682 with bad export cookie 11097586716329766806 [ 4308.059292] LustreError: 193087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4308.064355] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.557400] Lustre: server umount lustre-MDT0001 complete [ 4327.628260] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:38:21 (1782484701) [ 4332.521721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.745591] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4332.772561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4332.899201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.658624] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4341.733432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.825478] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4341.854476] Lustre: 199383:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4344.924133] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4349.709238] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4353.579763] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4358.357855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.486360] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4362.470118] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4365.906991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4367.754329] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4372.276430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4372.355067] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4377.205354] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4382.180843] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4385.773082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.783520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4385.788921] Lustre: Skipped 2 previous similar messages [ 4389.986030] Lustre: server umount lustre-OST0000 complete [ 4390.885458] LustreError: 201233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.894731] LustreError: 201233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4406.247037] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4406.250612] LustreError: Skipped 1 previous similar message [ 4408.003971] Lustre: server umount lustre-OST0001 complete [ 4416.173817] Lustre: server umount lustre-MDT0000 complete [ 4418.323017] LustreError: 199375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484792 with bad export cookie 11097586716329768213 [ 4418.323724] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4418.329852] LustreError: 199375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4418.505979] Lustre: server umount lustre-MDT0001 complete [ 4430.507833] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:40:04 (1782484804) [ 4437.758845] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4441.641860] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4456.643141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.740319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4456.758834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4456.803848] Lustre: lustre-MDT0000: new disk, initializing [ 4456.833293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.840155] Lustre: Skipped 3 previous similar messages [ 4456.847041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4458.654688] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4464.328842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.424097] Lustre: 207446:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4464.464685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4464.468724] Lustre: Skipped 1 previous similar message [ 4464.511093] Lustre: lustre-MDT0001: new disk, initializing [ 4464.570802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4464.574630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4466.718303] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4471.896873] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4474.833045] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4478.404337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4478.534532] Lustre: lustre-OST0000: new disk, initializing [ 4478.536738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4478.541307] Lustre: 209414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4481.363774] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4484.628698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4484.633757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4484.682550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4486.188189] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4488.027546] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4488.137205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4490.206912] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4490.301448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4494.816546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4494.820757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.825655] LustreError: Skipped 1 previous similar message [ 4494.829395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4494.833136] Lustre: Skipped 6 previous similar messages [ 4494.835403] Lustre: Skipped 5 previous similar messages [ 4497.307565] Lustre: server umount lustre-OST0000 complete [ 4505.517828] Lustre: server umount lustre-MDT0000 complete [ 4507.171065] LustreError: 207439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484881 with bad export cookie 11097586716329769774 [ 4507.174243] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4507.177854] LustreError: 207439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4507.318740] Lustre: server umount lustre-MDT0001 complete [ 4511.757456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.999671] LustreError: 211627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.011456] LustreError: 211627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4514.211583] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4522.438340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.817924] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4535.288864] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4542.704416] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4554.291154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4565.958970] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4574.664776] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4581.348677] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4581.556455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4587.123596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4587.504804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4593.632674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.663610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.664047] LustreError: Skipped 1 previous similar message [ 4593.715373] Lustre: Skipped 3 previous similar messages [ 4604.383745] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4604.684014] Lustre: server umount lustre-OST0000 complete [ 4610.926861] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4616.062329] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4644.349162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.556519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.562279] Lustre: Skipped 5 previous similar messages [ 4649.374846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4656.388120] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4658.933773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4658.952878] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4658.961669] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4659.001408] Lustre: Skipped 1 previous similar message [ 4663.490758] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 4669.432992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4669.433850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.465334] Lustre: Skipped 11 previous similar messages [ 4673.605907] Lustre: server umount lustre-OST0000 complete [ 4682.777635] Lustre: server umount lustre-MDT0000 complete [ 4684.770672] LustreError: 212689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.786883] LustreError: 212689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4685.886723] LustreError: 211608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485060 with bad export cookie 11097586716329770642 [ 4685.893921] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.200480] Lustre: server umount lustre-MDT0001 complete [ 4693.060814] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:44:26 (1782485066) [ 4698.391924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.805008] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4709.668691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.479505] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4719.659185] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4724.408722] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4729.407706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.167543] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4740.253232] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4744.100434] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4744.303880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4747.916978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4748.127753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4754.600273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.820168] Lustre: lustre-OST0001: new disk, initializing [ 4754.824629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4754.831381] Lustre: 220964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4760.429232] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4760.662703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4760.668776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4760.751797] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4768.130870] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4771.726406] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4771.892204] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4775.900379] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4776.095907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4781.543872] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4781.544437] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.553059] LustreError: Skipped 3 previous similar messages [ 4781.570318] Lustre: Skipped 5 previous similar messages [ 4783.566747] Lustre: server umount lustre-OST0001 complete [ 4788.499796] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4788.638819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4792.013596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4792.204636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4819.478257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.556361] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4825.078656] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4825.087510] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4825.089609] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4829.862879] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4831.222321] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4839.002659] Lustre: server umount lustre-OST0001 complete [ 4859.584948] Lustre: server umount lustre-OST0000 complete [ 4868.453149] Lustre: server umount lustre-MDT0000 complete [ 4871.662876] LustreError: 217575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485246 with bad export cookie 11097586716329771608 [ 4871.665627] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4871.670086] LustreError: 217575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4878.132162] Lustre: server umount lustre-MDT0001 complete [ 4899.696193] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:47:52 (1782485272) [ 4906.988748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.337809] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4907.377292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4907.585602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.588545] Lustre: Skipped 5 previous similar messages [ 4912.286340] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4920.408644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4920.512263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4920.543538] Lustre: 226203:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4925.170251] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4931.686533] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4936.504720] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4942.345392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4942.596767] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4948.645970] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4955.926515] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4964.094899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4964.177427] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4970.045756] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4977.182986] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4978.681164] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4982.276937] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4982.502918] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4986.443122] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4986.659914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4990.926084] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d8b44cfc800.ost_server_uuid 50 [ 4992.057153] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d8b44cfc800.ost_server_uuid in FULL state after 0 sec [ 4992.731926] Lustre: Permanently deactivating lustre-OST0001 [ 4992.757432] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4995.583308] Lustre: Permanently reactivating lustre-OST0001 [ 5003.750502] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5003.761378] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5003.764630] LustreError: Skipped 3 previous similar messages [ 5003.776748] Lustre: Skipped 16 previous similar messages [ 5005.013951] Lustre: server umount lustre-OST0001 complete [ 5008.869613] LustreError: 228049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.884102] LustreError: 228049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 5014.516996] Lustre: server umount lustre-OST0000 complete [ 5024.610833] Lustre: server umount lustre-MDT0000 complete [ 5027.615894] LustreError: 226194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485402 with bad export cookie 11097586716329772896 [ 5027.622568] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.625773] LustreError: 226194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5027.911155] Lustre: server umount lustre-MDT0001 complete [ 5049.464448] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:50:22 (1782485422) [ 5056.071338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5056.301518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5056.327071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5056.333414] Lustre: Skipped 5 previous similar messages [ 5060.055872] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5067.660068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.774328] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5067.793528] Lustre: 232892:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5071.661438] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5077.368556] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5081.452869] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5086.716238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.901810] Lustre: Permanently deactivating lustre-OST0000 [ 5086.911886] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5092.122909] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5096.509128] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5099.526759] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5104.952491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5105.046198] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5105.051932] Lustre: Skipped 1 previous similar message [ 5110.922627] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5115.026386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5118.128239] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5119.688976] Lustre: Permanently reactivating lustre-OST0000 [ 5119.714957] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5119.720508] Lustre: Skipped 2 previous similar messages [ 5123.504240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.519337] Lustre: Skipped 12 previous similar messages [ 5123.525506] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5123.535230] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.548538] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5123.552708] Lustre: Skipped 1 previous similar message [ 5123.591908] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5123.614377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5135.711844] Lustre: server umount lustre-OST0001 complete [ 5153.247379] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5153.456938] Lustre: server umount lustre-OST0000 complete [ 5163.389352] Lustre: server umount lustre-MDT0000 complete [ 5166.713166] LustreError: 232887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485541 with bad export cookie 11097586716329774541 [ 5166.717210] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5166.727676] LustreError: 232887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5166.973371] Lustre: server umount lustre-MDT0001 complete [ 5177.609383] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5180.019081] Key type lgssc unregistered [ 5180.224773] LNet: 237773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5180.229244] LNetError: 237773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5180.248878] LNet: Removed LNI 192.168.202.102@tcp [ 5180.890758] Key type .llcrypt unregistered [ 5180.894944] Key type ._llcrypt unregistered [ 5198.644542] Key type ._llcrypt registered [ 5198.646576] Key type .llcrypt registered [ 5198.739654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5199.585542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5199.748231] alg: No test for adler32 (adler32-zlib) [ 5200.819431] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 5201.032214] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5202.687446] Key type lgssc registered [ 5203.625466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5213.052269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5213.065141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.465774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.622571] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5223.912814] LustreError: 239219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.926469] LustreError: 239219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5226.714857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5226.904520] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5226.964600] Lustre: 239209:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5226.979911] Lustre: Permanently deactivating lustre-MDT0001 [ 5227.014244] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5227.317382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.845388] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5229.856366] LustreError: 239250:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5229.881573] LustreError: 239250:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5229.899996] LustreError: 239250:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 5231.697279] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5238.053944] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5242.960906] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5248.826207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5249.065599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5253.155846] LustreError: 241062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5253.182235] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5254.208646] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5258.213713] LustreError: 241061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.290668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5260.065651] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5264.361900] LustreError: 241062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.406242] LustreError: 241062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5265.993746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5266.233625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5271.550915] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5271.572273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5271.705189] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5277.404677] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5281.326711] Lustre: Permanently reactivating lustre-MDT0001 [ 5281.341318] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5281.344612] Lustre: Skipped 2 previous similar messages [ 5287.206263] Lustre: lustre-MDT0001: Client 448b8d5f-173d-4556-9f86-d77ecaffdb66 (at 192.168.202.2@tcp) reconnecting [ 5292.743362] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5294.491211] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5300.224604] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5300.495861] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5305.360548] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5305.756483] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5309.332069] Lustre: Permanently deactivating lustre-MDT0001 [ 5309.365392] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5309.374203] Lustre: Skipped 2 previous similar messages [ 5318.439708] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5329.893153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5329.900779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.928195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.976976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.994658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.020107] Lustre: Skipped 1 previous similar message [ 5334.976492] Lustre: server umount lustre-MDT0000 complete [ 5338.085768] LustreError: 239219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.133990] LustreError: 239219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5341.029631] LustreError: 239201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485715 with bad export cookie 5235237769060689180 [ 5341.039024] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5341.041096] LustreError: 239201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5341.617708] Lustre: server umount lustre-MDT0001 complete [ 5348.143320] Lustre: server umount lustre-OST0000 complete [ 5369.633508] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:55:42 (1782485742) [ 5386.974451] Lustre: server umount lustre-OST0001 complete [ 5395.237149] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 5407.757414] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5450.600335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.908493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5450.920083] Lustre: Skipped 2 previous similar messages [ 5450.954468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5451.068327] Lustre: lustre-MDT0000: new disk, initializing [ 5451.182343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.216222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5456.468949] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5469.496875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5469.632390] Lustre: 247502:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5469.722695] LustreError: 247514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5469.733110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5469.747531] Lustre: Skipped 1 previous similar message [ 5469.857666] Lustre: lustre-MDT0001: new disk, initializing [ 5469.994196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5470.044769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5470.063717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5474.710750] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5485.001859] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5492.026222] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5501.050728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5501.499822] Lustre: lustre-OST0000: new disk, initializing [ 5501.508599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5501.522096] Lustre: 249472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5501.699123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.639994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5507.651758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5507.895053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5509.812700] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5520.939980] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5523.419058] LustreError: 247510:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5532.685957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5532.943265] Lustre: lustre-OST0001: new disk, initializing [ 5532.960897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5532.975895] Lustre: 250586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5533.447163] LustreError: 247510:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5533.489487] LustreError: 250359:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5539.190969] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5540.470546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5540.494143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5540.636550] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5543.553031] LustreError: 250359:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5550.337091] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5552.610996] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5556.198560] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.205471] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5556.207374] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5556.207383] Lustre: Skipped 3 previous similar messages [ 5556.217748] Lustre: Skipped 2 previous similar messages [ 5561.329720] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5561.345054] Lustre: Skipped 2 previous similar messages [ 5561.732645] Lustre: server umount lustre-OST0001 complete [ 5566.446771] LustreError: 249455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.463347] LustreError: 249455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5579.744232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5579.747397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.750151] LustreError: Skipped 1 previous similar message [ 5579.751109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5579.781570] Lustre: Skipped 1 previous similar message [ 5585.536163] Lustre: server umount lustre-OST0000 complete [ 5592.032585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.051576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5592.058656] Lustre: Skipped 4 previous similar messages [ 5604.319177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5606.589852] Lustre: server umount lustre-MDT0000 complete [ 5607.392505] LustreError: 247514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.425414] LustreError: 247514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5611.315675] LustreError: 248297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485985 with bad export cookie 5235237769060691994 [ 5611.327170] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5611.340219] LustreError: 248297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5617.997637] Lustre: server umount lustre-MDT0001 complete [ 5631.208915] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5634.555032] Key type lgssc unregistered [ 5634.883855] LNet: 252622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5634.888690] LNetError: 252622:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5634.902778] LNet: Removed LNI 192.168.202.102@tcp [ 5635.860553] Key type .llcrypt unregistered [ 5635.863553] Key type ._llcrypt unregistered [ 5690.036803] Key type ._llcrypt registered [ 5690.044132] Key type .llcrypt registered [ 5690.254272] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5692.236134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5692.278107] alg: No test for adler32 (adler32-zlib) [ 5693.394746] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 5693.719041] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5695.465136] Key type lgssc registered [ 5697.337792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5706.816605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.867599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.316903] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5708.340314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5708.507511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.201785] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5723.639684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.725059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5723.756808] Lustre: 254848:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5724.184038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.444642] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5737.308561] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5743.682107] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5751.845345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5752.129372] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5752.299790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5757.658236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5759.127713] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5766.922487] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5819.365226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.365400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.366412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.372078] Lustre: Skipped 1 previous similar message [ 5824.472726] Lustre: server umount lustre-OST0000 complete [ 5828.470090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5839.365638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.688881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.048885] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5851.697994] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5859.054625] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.063721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.065565] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5859.075094] Lustre: Skipped 1 previous similar message [ 5890.018896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5890.021324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.032649] LustreError: Skipped 2 previous similar messages [ 5890.038239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5890.048073] Lustre: Skipped 1 previous similar message [ 5890.062127] Lustre: Skipped 2 previous similar messages [ 5895.040657] Lustre: server umount lustre-OST0000 complete [ 5900.257992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.260372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.281167] Lustre: Skipped 1 previous similar message [ 5900.300163] Lustre: Skipped 1 previous similar message [ 5905.376390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.381422] Lustre: Skipped 1 previous similar message [ 5910.497727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5910.507264] Lustre: Skipped 1 previous similar message [ 5913.055237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5913.166208] Lustre: server umount lustre-MDT0000 complete [ 5916.009862] LustreError: 254839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486290 with bad export cookie 15844024026971499717 [ 5916.015904] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5916.017050] LustreError: 254839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5916.336659] Lustre: server umount lustre-MDT0001 complete [ 5925.774939] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 5928.161105] Key type lgssc unregistered [ 5928.439266] LNet: 260268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5928.448850] LNetError: 260268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5929.517143] LNet: Removed LNI 192.168.202.102@tcp [ 5930.276893] Key type .llcrypt unregistered [ 5930.278721] Key type ._llcrypt unregistered [ 5951.863439] Key type ._llcrypt registered [ 5951.867238] Key type .llcrypt registered [ 5952.011922] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 5953.087973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5953.116532] alg: No test for adler32 (adler32-zlib) [ 5954.091197] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 5954.314038] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 5955.951471] Key type lgssc registered [ 5956.951436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.207585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5964.225856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.626274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.577735] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5976.037951] LustreError: 261686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.077758] LustreError: 261686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5976.976328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.285231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5981.702868] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 5988.147375] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5992.671542] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5998.929210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5999.271958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6003.866080] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6008.677963] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6010.539761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6031.332161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.332395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6031.336604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.342148] Lustre: Skipped 1 previous similar message [ 6035.205409] Lustre: server umount lustre-OST0000 complete [ 6039.007938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6039.015980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.035478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.044810] Lustre: Skipped 1 previous similar message [ 6041.570475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.583679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6041.589907] Lustre: Skipped 1 previous similar message [ 6046.696973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6046.702794] Lustre: Skipped 1 previous similar message [ 6050.792445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.802404] Lustre: Skipped 1 previous similar message [ 6052.319178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6052.471661] Lustre: server umount lustre-MDT0000 complete [ 6055.468335] LustreError: 261667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486429 with bad export cookie 3900285515302151276 [ 6055.479572] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.483168] LustreError: 261667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6055.833844] Lustre: server umount lustre-MDT0001 complete [ 6064.004904] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6066.178764] Key type lgssc unregistered [ 6066.434816] LNet: 265990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6066.443266] LNetError: 265990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6067.497748] LNet: Removed LNI 192.168.202.102@tcp [ 6068.251211] Key type .llcrypt unregistered [ 6068.253638] Key type ._llcrypt unregistered [ 6083.326514] Key type ._llcrypt registered [ 6083.329189] Key type .llcrypt registered [ 6083.402198] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6084.186454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6084.196087] alg: No test for adler32 (adler32-zlib) [ 6085.200941] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 6085.416796] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6087.055161] Key type lgssc registered [ 6087.957510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6094.191333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6094.224730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.638574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.727848] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6106.085256] LustreError: 267229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.102968] LustreError: 267229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6106.326483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.612457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6109.736169] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6115.124708] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6119.147112] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6124.515818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6124.805651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.461196] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6135.331298] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6141.243668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6156.769025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.775606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.780973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.799824] Lustre: Skipped 1 previous similar message [ 6159.538876] Lustre: server umount lustre-OST0000 complete [ 6162.912148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6162.917712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.935052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.937616] Lustre: Skipped 1 previous similar message [ 6167.008674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.010957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6167.019040] Lustre: Skipped 1 previous similar message [ 6168.580828] Lustre: server umount lustre-MDT0000 complete [ 6171.372571] LustreError: 267209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486545 with bad export cookie 849645880519268711 [ 6171.374508] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6171.396767] LustreError: 267209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6171.659843] Lustre: server umount lustre-MDT0001 complete [ 6180.261403] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6182.273223] Key type lgssc unregistered [ 6182.506796] LNet: 270923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6182.511707] LNetError: 270923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6182.525554] LNet: Removed LNI 192.168.202.102@tcp [ 6183.147321] Key type .llcrypt unregistered [ 6183.149443] Key type ._llcrypt unregistered [ 6201.201752] Key type ._llcrypt registered [ 6201.204753] Key type .llcrypt registered [ 6201.292771] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6202.106675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6202.137063] alg: No test for adler32 (adler32-zlib) [ 6203.154112] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 6203.356052] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6205.039181] Key type lgssc registered [ 6206.073839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6211.495162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6211.521796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.813505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6215.738131] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6222.126304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.444573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.297547] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6232.147929] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6236.250155] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6241.068163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.256137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.448952] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6250.188029] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6257.644564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6278.118148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6278.134669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.150516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.155951] Lustre: Skipped 1 previous similar message [ 6283.236246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.247590] Lustre: Skipped 2 previous similar messages [ 6283.849774] Lustre: server umount lustre-OST0000 complete [ 6288.359526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.374994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6288.379599] Lustre: Skipped 1 previous similar message [ 6288.401801] Lustre: Skipped 1 previous similar message [ 6293.477778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.489341] Lustre: Skipped 1 previous similar message [ 6295.065690] Lustre: server umount lustre-MDT0000 complete [ 6298.591965] LustreError: 272319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.616059] LustreError: 272319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6298.895918] LustreError: 272300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486673 with bad export cookie 15976920035914677542 [ 6298.898616] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.904085] LustreError: 272300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6299.586643] Lustre: server umount lustre-MDT0001 complete [ 6311.910297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6317.182497] Key type lgssc unregistered [ 6317.940226] LNet: 276645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6317.961106] LNetError: 276645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6319.015601] LNet: Removed LNI 192.168.202.102@tcp [ 6320.555064] Key type .llcrypt unregistered [ 6320.556734] Key type ._llcrypt unregistered [ 6345.458259] Key type ._llcrypt registered [ 6345.461723] Key type .llcrypt registered [ 6345.650553] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6348.042191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6348.098434] alg: No test for adler32 (adler32-zlib) [ 6349.158826] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 6349.439791] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6351.151196] Key type lgssc registered [ 6352.304264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6358.932821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6358.948530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.587055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.417581] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6370.786056] LustreError: 277882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.799200] LustreError: 277882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6373.503584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.809115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6377.935970] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6385.318246] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6390.552128] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6396.900359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.215994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.907582] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6406.465607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6410.171867] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6421.985421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6421.994203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6422.003276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6422.017426] Lustre: Skipped 1 previous similar message [ 6422.038804] Lustre: Skipped 1 previous similar message [ 6426.232840] Lustre: server umount lustre-OST0000 complete [ 6430.179216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6430.188141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6430.199699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6432.226030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6432.233864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.059779] Lustre: server umount lustre-MDT0000 complete [ 6437.345381] LustreError: 277884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.364057] LustreError: 277884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6439.384329] LustreError: 277849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486813 with bad export cookie 17949594947488768916 [ 6439.401807] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6439.725676] Lustre: server umount lustre-MDT0001 complete [ 6448.385200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6450.768410] Key type lgssc unregistered [ 6451.004781] LNet: 281580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6451.008146] LNetError: 281580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6452.071943] LNet: Removed LNI 192.168.202.102@tcp [ 6452.849197] Key type .llcrypt unregistered [ 6452.851070] Key type ._llcrypt unregistered [ 6463.571390] Key type ._llcrypt registered [ 6463.575116] Key type .llcrypt registered [ 6463.726267] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6478.085340] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6490.048460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6490.871578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6490.899352] alg: No test for adler32 (adler32-zlib) [ 6492.003841] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 6492.201877] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6493.928548] Key type lgssc registered [ 6495.241968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6525.971201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6525.988875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6527.275612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6527.301569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6527.428694] Lustre: lustre-MDT0000: new disk, initializing [ 6527.510922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6527.528484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6531.605499] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6541.066667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.195700] Lustre: 285112:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6541.295581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6541.300445] Lustre: Skipped 1 previous similar message [ 6541.376548] Lustre: lustre-MDT0001: new disk, initializing [ 6541.452873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.489700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6541.495907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6545.436619] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6553.138385] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6557.888580] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6564.024831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6564.378345] Lustre: lustre-OST0000: new disk, initializing [ 6564.383189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6564.390740] Lustre: 287078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6564.508624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.054668] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6572.104380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6572.117600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6572.234545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6578.722853] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6583.152717] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6583.307314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6587.154572] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6587.368253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6592.993618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.995672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6593.005977] LustreError: Skipped 1 previous similar message [ 6593.011048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6593.022245] Lustre: Skipped 1 previous similar message [ 6593.036911] Lustre: Skipped 1 previous similar message [ 6596.064939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.072047] Lustre: Skipped 1 previous similar message [ 6601.187417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6601.192391] Lustre: Skipped 1 previous similar message [ 6603.232078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6603.355606] Lustre: server umount lustre-OST0000 complete [ 6606.305440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.325245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.426158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.428289] Lustre: Skipped 2 previous similar messages [ 6612.305754] Lustre: server umount lustre-MDT0000 complete [ 6615.329861] LustreError: 287077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486989 with bad export cookie 12681836154844295789 [ 6615.333183] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.336591] LustreError: 287077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6615.561305] Lustre: server umount lustre-MDT0001 complete [ 6621.665540] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:16:35 (1782486995) [ 6626.729891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.242080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6631.379075] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6632.416984] LustreError: 289491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6632.424132] LustreError: 289491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6637.559443] LustreError: 289490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.350585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.675023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6642.252431] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6648.432738] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6653.600929] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6659.130205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.418922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.852295] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6671.003979] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6688.241850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.253362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6688.254683] Lustre: Skipped 2 previous similar messages [ 6688.266568] Lustre: Skipped 1 previous similar message [ 6690.747652] Lustre: server umount lustre-OST0000 complete [ 6694.881042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6694.885764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.073349] Lustre: server umount lustre-MDT0000 complete [ 6702.973375] LustreError: 289471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487077 with bad export cookie 12681836154844296678 [ 6702.979664] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6702.985568] LustreError: 289471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6703.189321] Lustre: server umount lustre-MDT0001 complete [ 6710.815455] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 6712.592331] Key type lgssc unregistered [ 6712.790607] LNet: 293049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6712.793722] LNetError: 293049:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6712.810690] LNet: Removed LNI 192.168.202.102@tcp [ 6713.454756] Key type .llcrypt unregistered [ 6713.457653] Key type ._llcrypt unregistered [ 6734.274488] Key type ._llcrypt registered [ 6734.276887] Key type .llcrypt registered [ 6734.369387] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6744.794914] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6745.604318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6745.638469] alg: No test for adler32 (adler32-zlib) [ 6746.630945] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 6746.825984] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 6748.463201] Key type lgssc registered [ 6749.134769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6778.300804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6778.317473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.450713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6779.466411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6779.516226] Lustre: lustre-MDT0000: new disk, initializing [ 6779.564643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6779.572568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6782.538519] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6790.300763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.423938] Lustre: 296851:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6790.477696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6790.482256] Lustre: Skipped 1 previous similar message [ 6790.536631] Lustre: lustre-MDT0001: new disk, initializing [ 6790.607780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6790.642621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6790.652121] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6794.077982] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6801.204109] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6805.615488] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6810.444809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.647705] Lustre: lustre-OST2710: new disk, initializing [ 6810.650804] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6810.656718] Lustre: 298818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6810.746158] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6815.323512] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6815.788590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6815.793220] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6815.889662] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6821.975117] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6826.439420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6826.562052] Lustre: lustre-OST03e8: new disk, initializing [ 6826.566052] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6826.572221] Lustre: 299883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6826.643354] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6830.423804] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6832.685867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6832.699392] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6832.761813] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6835.150044] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6839.261088] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6839.388251] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6842.222501] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6842.346379] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6844.860755] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6845.010555] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6847.674477] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6847.825414] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6852.064690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6852.069264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.075850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.092870] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6853.093971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.232075] Lustre: server umount lustre-MDT0000 complete [ 6858.427803] LustreError: 296843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487232 with bad export cookie 3928740250029318544 [ 6858.434109] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6858.441779] LustreError: 296843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6858.683830] Lustre: server umount lustre-MDT0001 complete [ 6871.586519] Lustre: server umount lustre-OST2710 complete [ 6884.109110] Lustre: server umount lustre-OST03e8 complete [ 6895.534245] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 6900.826364] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6921.089155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.202497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6921.227650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6921.283718] Lustre: lustre-MDT0000: new disk, initializing [ 6921.328360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.337161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6923.790222] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6929.894075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.946833] Lustre: 305124:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6929.983105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6929.986350] Lustre: Skipped 1 previous similar message [ 6930.030741] Lustre: lustre-MDT0001: new disk, initializing [ 6930.101019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6930.130192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6930.136315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6932.590251] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6938.466557] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6941.501167] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6945.361539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.494912] Lustre: 307088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6945.699048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6945.770902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6949.015573] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 6954.883507] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6957.437203] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6957.541378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6959.775427] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6959.909582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6961.123847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.124807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6961.137102] Lustre: Skipped 2 previous similar messages [ 6961.138924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6961.138929] Lustre: Skipped 3 previous similar messages [ 6966.242118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.245046] Lustre: Skipped 1 previous similar message [ 6967.211791] Lustre: server umount lustre-OST0000 complete [ 6971.361228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.363500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6971.363927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6971.363931] Lustre: Skipped 1 previous similar message [ 6971.380507] Lustre: Skipped 2 previous similar messages [ 6975.425578] Lustre: server umount lustre-MDT0000 complete [ 6976.481909] LustreError: 305133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.501405] LustreError: 305133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6977.473643] LustreError: 305115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487351 with bad export cookie 3928740250029320217 [ 6977.475321] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6977.485341] LustreError: 305115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6977.693704] Lustre: server umount lustre-MDT0001 complete [ 6982.193348] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:22:36 (1782487356) [ 6982.968083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6983.823823] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:22:37 (1782487357) [ 6985.394194] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 6998.786202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6998.888348] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6998.899411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6998.969373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6998.973150] Lustre: Skipped 1 previous similar message [ 7001.086087] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7005.702030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.772876] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7005.789454] Lustre: 310943:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7008.064892] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7025.833502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7025.947627] Lustre: Denying initial registration attempt from nid 192.168.202.102@tcp, specified as failover [ 7025.956247] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7025.964796] LustreError: 312400:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 7025.972586] LustreError: 312400:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7025.976568] LustreError: 312400:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7026.012961] Lustre: server umount lustre-OST0000 complete [ 7026.015167] LustreError: 312400:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7031.776090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7031.780660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.792139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7031.794306] Lustre: Skipped 1 previous similar message [ 7035.826961] Lustre: server umount lustre-MDT0000 complete [ 7037.409456] LustreError: 310954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.416831] LustreError: 310954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7037.641535] LustreError: 310935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487412 with bad export cookie 3928740250029321099 [ 7037.646265] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7037.648683] LustreError: 310935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7037.824256] Lustre: server umount lustre-MDT0001 complete [ 7042.734983] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7043.946606] Key type lgssc unregistered [ 7044.117704] LNet: 313519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7044.121542] LNetError: 313519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7044.130506] LNet: Removed LNI 192.168.202.102@tcp [ 7044.516096] Key type .llcrypt unregistered [ 7044.517453] Key type ._llcrypt unregistered [ 7049.724689] Key type ._llcrypt registered [ 7049.726389] Key type .llcrypt registered [ 7049.780561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7050.196809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7050.241096] alg: No test for adler32 (adler32-zlib) [ 7051.152719] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 7051.282864] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7052.895199] Key type lgssc registered [ 7053.338180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7082.125369] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7086.581161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7086.592521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.716305] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7087.729957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7087.805740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.868184] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7094.377024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.442526] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7094.456867] Lustre: 316257:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7094.596627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.556661] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7100.277965] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7103.207417] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7120.420238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7120.534821] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7120.641135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.702926] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7128.319980] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7129.665561] Lustre: server umount lustre-OST0000 complete [ 7135.203246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.204315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7135.214461] Lustre: Skipped 1 previous similar message [ 7137.590061] Lustre: server umount lustre-MDT0000 complete [ 7139.544602] LustreError: 316248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487514 with bad export cookie 17985648521839831894 [ 7139.549896] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7139.551919] LustreError: 316248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7139.713977] Lustre: server umount lustre-MDT0001 complete [ 7145.394581] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7147.053158] Key type lgssc unregistered [ 7147.239574] LNet: 319826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7147.246563] LNetError: 319826:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7147.260683] LNet: Removed LNI 192.168.202.102@tcp [ 7147.821154] Key type .llcrypt unregistered [ 7147.823772] Key type ._llcrypt unregistered [ 7161.483885] Key type ._llcrypt registered [ 7161.485949] Key type .llcrypt registered [ 7161.546816] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7162.116402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7162.129905] alg: No test for adler32 (adler32-zlib) [ 7163.008791] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 7163.119978] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7164.711171] Key type lgssc registered [ 7165.256155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7168.996135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7169.006212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.206899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.175907] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7176.613118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.860824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7178.950974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7183.109455] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7186.151791] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7191.652699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7191.828479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.381409] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7197.493027] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7210.248205] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 7212.514135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.515790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7212.518200] Lustre: Skipped 2 previous similar messages [ 7212.522019] Lustre: Skipped 2 previous similar messages [ 7214.167600] Lustre: server umount lustre-MDT0000 complete [ 7215.348615] LustreError: 323708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.354540] LustreError: 323708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7215.583404] LustreError: 321200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487590 with bad export cookie 2972872246370479513 [ 7215.586222] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7215.588156] LustreError: 321200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7217.634030] LustreError: 321216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.634552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.650980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7220.470348] LustreError: 321214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.482708] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 7221.747982] Lustre: server umount lustre-MDT0001 complete [ 7224.325746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7227.670547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.186811] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2941c38e9f87c98b [ 7241.191525] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [ 7241.320717] LustreError: 323048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7241.329144] LustreError: 323048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7241.360049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.390791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7243.196103] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7246.497250] LustreError: 323049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.503054] LustreError: 323049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7246.833212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.998679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7248.683639] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7251.108281] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7251.837866] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7252.510168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7254.609535] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7261.317440] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 6 sec [ 7266.786388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.787351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7266.791858] Lustre: Skipped 1 previous similar message [ 7266.795830] Lustre: Skipped 2 previous similar messages [ 7277.023213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7277.025429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7277.033192] Lustre: Skipped 3 previous similar messages [ 7277.094620] Lustre: server umount lustre-OST0000 complete [ 7282.144738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.149555] Lustre: Skipped 1 previous similar message [ 7284.761294] Lustre: server umount lustre-MDT0000 complete [ 7286.410388] LustreError: 321201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487660 with bad export cookie 2972872246370486667 [ 7286.413815] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7286.415032] LustreError: 321201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7286.568541] Lustre: server umount lustre-MDT0001 complete [ 7291.186049] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7292.394876] Key type lgssc unregistered [ 7292.528463] LNet: 327481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7292.530882] LNetError: 327481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.540327] LNet: Removed LNI 192.168.202.102@tcp [ 7292.915651] Key type .llcrypt unregistered [ 7292.917243] Key type ._llcrypt unregistered [ 7304.305023] Key type ._llcrypt registered [ 7304.306068] Key type .llcrypt registered [ 7304.363097] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7304.855899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7304.873982] alg: No test for adler32 (adler32-zlib) [ 7305.756253] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 7305.875318] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7307.471168] Key type lgssc registered [ 7308.004876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7312.019949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7312.037957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.244230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.395466] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7319.434068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.608598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7321.463574] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7324.508284] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7326.967810] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7329.760500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7329.764154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7329.769733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.787546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.787919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.440857] Lustre: server umount lustre-MDT0000 complete [ 7335.906483] LustreError: 329605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.913276] LustreError: 329605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7335.950889] LustreError: 328858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487710 with bad export cookie 5782120536513579972 [ 7335.952258] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.956057] LustreError: 328858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7336.095713] Lustre: server umount lustre-MDT0001 complete [ 7342.471948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.600613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7342.616870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7342.723898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7344.557871] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7348.660353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.725248] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7348.743194] Lustre: 331461:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7348.879279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7350.705275] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7354.212804] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7356.427317] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7361.702934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.800389] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7361.806134] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7361.810904] LustreError: 333343:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7361.814061] LustreError: 333343:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7361.817059] LustreError: 333343:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7361.847240] Lustre: server umount lustre-OST0000 complete [ 7361.849976] LustreError: 333343:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7367.654732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.801768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7367.910856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.456334] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7377.116556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7386.575669] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7398.500227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.663994] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7398.994038] Lustre: lustre-OST0001: new disk, initializing [ 7399.000737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7399.012646] Lustre: 334885:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7399.209716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7405.792240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7405.796811] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7406.083592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7408.331943] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7419.909489] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7426.541395] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7426.546893] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.595756] LustreError: Skipped 1 previous similar message [ 7426.601213] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7426.602034] Lustre: Skipped 1 previous similar message [ 7426.608915] Lustre: Skipped 1 previous similar message [ 7431.771406] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7431.789973] Lustre: Skipped 1 previous similar message [ 7437.302060] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7437.321186] Lustre: Skipped 1 previous similar message [ 7439.842174] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7440.321561] Lustre: server umount lustre-OST0001 complete [ 7442.410353] LustreError: 334332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.454719] LustreError: 334332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7447.525916] LustreError: 334332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.528062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.562555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7447.570615] LustreError: 334332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7447.649905] Lustre: Skipped 1 previous similar message [ 7452.651700] LustreError: 333809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7452.676854] LustreError: 333809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7457.772582] LustreError: 333809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7457.798414] LustreError: 333809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7460.320431] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7460.606419] Lustre: server umount lustre-OST0000 complete [ 7467.489214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7467.495726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7467.505476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7467.515035] Lustre: Skipped 5 previous similar messages [ 7470.998939] Lustre: server umount lustre-MDT0000 complete [ 7473.121652] LustreError: 331469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.136682] LustreError: 331469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7475.888242] LustreError: 331454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487850 with bad export cookie 5782120536513580525 [ 7475.889687] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7475.909702] LustreError: 331454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7476.720964] Lustre: server umount lustre-MDT0001 complete [ 7492.544604] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7496.333517] Key type lgssc unregistered [ 7496.690668] LNet: 336848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7496.695850] LNetError: 336848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7496.712813] LNet: Removed LNI 192.168.202.102@tcp [ 7497.718289] Key type .llcrypt unregistered [ 7497.721975] Key type ._llcrypt unregistered [ 7575.906473] Key type ._llcrypt registered [ 7575.918537] Key type .llcrypt registered [ 7576.225940] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7594.444153] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7595.770994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7595.805584] alg: No test for adler32 (adler32-zlib) [ 7596.889476] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 7597.220369] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7598.912299] Key type lgssc registered [ 7600.354532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7638.481370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7638.512451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7639.906646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7639.970572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7640.113923] Lustre: lustre-MDT0000: new disk, initializing [ 7640.202612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.221859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7645.220155] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7656.264337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7656.431636] Lustre: 341784:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7656.531892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7656.536234] Lustre: Skipped 1 previous similar message [ 7656.679741] Lustre: lustre-MDT0001: new disk, initializing [ 7656.835906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7656.893226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7656.918417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7661.171810] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7671.707888] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7677.009363] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7685.014791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.633831] Lustre: lustre-OST0000: new disk, initializing [ 7685.639927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7685.650735] Lustre: 343756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7685.823785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7691.918289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7691.938530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7692.143277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7695.373795] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7709.974807] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7716.635890] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7717.109320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7724.434753] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7724.798354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7728.099925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7728.117918] LustreError: Skipped 1 previous similar message [ 7728.118019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7728.128966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7728.166377] Lustre: Skipped 1 previous similar message [ 7733.397440] Lustre: server umount lustre-OST0000 complete [ 7738.336828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.337819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7738.353984] Lustre: Skipped 1 previous similar message [ 7738.371057] Lustre: Skipped 2 previous similar messages [ 7743.459244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7744.470944] Lustre: server umount lustre-MDT0000 complete [ 7748.517031] LustreError: 341776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488123 with bad export cookie 4476525864785726161 [ 7748.525515] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7748.529952] LustreError: 341776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7749.018663] Lustre: server umount lustre-MDT0001 complete [ 7757.673880] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:35:30 (1782488130) [ 7765.873955] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:35:39 (1782488139) [ 7775.192852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7775.531708] LustreError: 346512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.547531] LustreError: 346512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7775.599465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.824384] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7780.834280] LustreError: 346513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.954778] LustreError: 346512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7790.383880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.000563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7797.084418] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7806.635623] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7814.946244] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7823.971180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7824.463307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.980783] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7839.510035] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7853.290097] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7855.279450] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7857.527948] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7859.244539] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7860.966045] Lustre: Failing over lustre-MDT0000 [ 7861.266418] Lustre: server umount lustre-MDT0000 complete [ 7861.522198] LustreError: 346509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.730394] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.750222] Lustre: Skipped 2 previous similar messages [ 7866.627129] LustreError: 346509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.655828] LustreError: 346509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7876.856581] LustreError: 346509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.886239] LustreError: 346509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7878.111468] Lustre: 340008:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488236/real 1782488236] req@00000000a4cde399 x1869074106756096/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782488252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7878.129976] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7881.429451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.738391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.831095] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7893.242107] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7893.939617] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7893.999653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7894.031333] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7894.117687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7904.396878] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7906.189996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7910.530669] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7912.988255] Lustre: Failing over lustre-MDT0000 [ 7913.454441] Lustre: server umount lustre-MDT0000 complete [ 7913.783982] LustreError: 346507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.814793] LustreError: 346507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 7914.483209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.488229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7929.761575] Lustre: 340008:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488288/real 1782488288] req@00000000e9ce2f94 x1869074106777216/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782488304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7929.778442] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.078275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.309633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7940.362699] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7944.441634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7945.115901] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7945.726183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7945.731048] Lustre: Skipped 2 previous similar messages [ 7945.759298] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7945.827784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7955.031568] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7956.694245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7966.181436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.184410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7966.190040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7966.190052] Lustre: Skipped 1 previous similar message [ 7966.192253] Lustre: Skipped 3 previous similar messages [ 7969.432828] Lustre: server umount lustre-MDT0000 complete [ 7971.305705] LustreError: 347447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.320177] LustreError: 347447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7973.135552] LustreError: 346492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488347 with bad export cookie 4476525864785728877 [ 7973.146897] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.151134] LustreError: 346492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7973.526920] Lustre: server umount lustre-MDT0001 complete [ 7987.075729] Lustre: server umount lustre-OST0000 complete [ 8010.415306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.904370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.498845] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8031.232754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8037.387442] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8045.882693] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8053.342349] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8064.467617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8073.626320] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8082.444641] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8082.489583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8087.010153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8087.023288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.033646] Lustre: Skipped 1 previous similar message [ 8087.047098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8087.062588] Lustre: Skipped 2 previous similar messages [ 8088.945576] Lustre: server umount lustre-OST0000 complete [ 8097.807681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.816214] Lustre: Skipped 2 previous similar messages [ 8100.015720] Lustre: server umount lustre-MDT0000 complete [ 8102.885201] LustreError: 352877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.899048] LustreError: 352877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8103.422430] LustreError: 352858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488477 with bad export cookie 4476525864785729619 [ 8103.441117] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8103.827558] Lustre: server umount lustre-MDT0001 complete [ 8114.774759] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8117.235571] Key type lgssc unregistered [ 8117.572073] LNet: 356412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8117.585204] LNetError: 356412:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8118.650420] LNet: Removed LNI 192.168.202.102@tcp [ 8119.490291] Key type .llcrypt unregistered [ 8119.492550] Key type ._llcrypt unregistered [ 8141.518317] Key type ._llcrypt registered [ 8141.520670] Key type .llcrypt registered [ 8141.590908] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8142.315173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8142.435103] alg: No test for adler32 (adler32-zlib) [ 8143.462629] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8143.659803] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8145.287176] Key type lgssc registered [ 8146.153614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.533534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8151.547827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.991968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.128179] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8161.983564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.189915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8164.573871] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8169.204089] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8172.698985] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8176.390401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.638749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.644331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8180.324427] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8184.605162] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8197.088500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.095294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.099140] Lustre: Skipped 2 previous similar messages [ 8197.105435] Lustre: Skipped 1 previous similar message [ 8201.371424] Lustre: server umount lustre-MDT0000 complete [ 8202.209688] LustreError: 360400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.221296] LustreError: 360400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8203.508384] LustreError: 357890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488577 with bad export cookie 14129493928635712256 [ 8203.514140] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8203.516206] LustreError: 357890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8203.723085] Lustre: server umount lustre-MDT0001 complete [ 8214.621724] Lustre: server umount lustre-OST0000 complete [ 8219.285861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.442133] LustreError: 361618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.466883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.843111] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8223.672676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.795814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.161851] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8227.545711] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8229.317559] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8231.614487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8233.573750] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8235.897598] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8245.029969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8254.432343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8254.434684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.438525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.456659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.457062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8255.458939] Lustre: Skipped 1 previous similar message [ 8255.463697] Lustre: Skipped 1 previous similar message [ 8259.542728] Lustre: server umount lustre-MDT0000 complete [ 8260.576160] LustreError: 361619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.583422] LustreError: 361619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8260.808395] LustreError: 361598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488635 with bad export cookie 14129493928635713369 [ 8260.810593] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.812222] LustreError: 361598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8260.945838] Lustre: server umount lustre-MDT0001 complete [ 8272.475086] Lustre: server umount lustre-OST0000 complete [ 8277.395533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8277.538429] LustreError: 365545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8277.562036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.564018] Lustre: Skipped 1 previous similar message [ 8278.825978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8281.646272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.087974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8285.503377] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8287.301935] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8289.576667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.805850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8294.285498] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8303.013181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8307.167950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8307.169813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.173876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8307.175558] Lustre: Skipped 1 previous similar message [ 8308.193246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8312.284774] Lustre: server umount lustre-OST0000 complete [ 8317.919782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8317.924216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.928052] Lustre: Skipped 1 previous similar message [ 8317.930420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8317.933330] Lustre: Skipped 2 previous similar messages [ 8319.829938] Lustre: server umount lustre-MDT0000 complete [ 8321.171772] LustreError: 368044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488695 with bad export cookie 14129493928635714377 [ 8321.175586] LustreError: 368044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8321.175892] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8321.290917] Lustre: server umount lustre-MDT0001 complete [ 8324.975134] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8326.111209] Key type lgssc unregistered [ 8326.243347] LNet: 369097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8326.245939] LNetError: 369097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8326.254300] LNet: Removed LNI 192.168.202.102@tcp [ 8326.587747] Key type .llcrypt unregistered [ 8326.589508] Key type ._llcrypt unregistered [ 8336.419492] Key type ._llcrypt registered [ 8336.420781] Key type .llcrypt registered [ 8336.460403] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8336.776699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8336.838024] alg: No test for adler32 (adler32-zlib) [ 8337.702488] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8337.790034] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8339.367149] Key type lgssc registered [ 8339.760333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8342.546988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8342.553880] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.557388] LustreError: 370514:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8342.566478] LustreError: 370514:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8342.570084] LustreError: 370514:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8342.573713] LustreError: 370514:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8342.577675] LustreError: 370514:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8345.050410] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.053395] LustreError: 370911:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8345.061970] LustreError: 370911:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8345.064223] LustreError: 370911:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8345.066368] LustreError: 370911:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8345.068869] LustreError: 370911:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8351.565387] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8352.643676] Key type lgssc unregistered [ 8352.779207] LNet: 371911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8352.781399] LNetError: 371911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8352.789548] LNet: Removed LNI 192.168.202.102@tcp [ 8353.117412] Key type .llcrypt unregistered [ 8353.118405] Key type ._llcrypt unregistered [ 8359.238652] Key type ._llcrypt registered [ 8359.239737] Key type .llcrypt registered [ 8359.277804] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8365.610800] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8365.995939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8366.005151] alg: No test for adler32 (adler32-zlib) [ 8366.869770] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8366.978516] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8368.599255] Key type lgssc registered [ 8369.231649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8393.988935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8394.005145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.188658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8395.211973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8395.283277] Lustre: lustre-MDT0000: new disk, initializing [ 8395.345487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8395.362315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8398.591486] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8410.141280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.301799] Lustre: 375043:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8410.430505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8410.433710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8410.439386] Lustre: Skipped 1 previous similar message [ 8410.516894] Lustre: lustre-MDT0001: new disk, initializing [ 8410.674689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8410.736265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8410.756270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8416.561815] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8429.839307] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8441.773649] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8453.445668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8454.051378] Lustre: lustre-OST0000: new disk, initializing [ 8454.060148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8454.073815] Lustre: 377019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8454.323126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8460.513846] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8462.382937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8462.393381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8462.459918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8467.463213] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8470.541808] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8470.710962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8473.730850] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8473.859960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8477.665803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8477.672951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8477.695276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.700191] Lustre: Skipped 1 previous similar message [ 8481.285227] Lustre: server umount lustre-OST0000 complete [ 8487.392678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8487.397987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.406141] Lustre: Skipped 1 previous similar message [ 8487.409631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.780932] Lustre: server umount lustre-MDT0000 complete [ 8492.262743] LustreError: 375035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488866 with bad export cookie 8957356739374861505 [ 8492.268936] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8492.273497] LustreError: 375035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8492.493566] Lustre: server umount lustre-MDT0001 complete [ 8497.915733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8499.020153] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:47:52 (1782488872) [ 8503.386915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.705399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.250816] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8508.899338] LustreError: 379477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8508.926317] LustreError: 379477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8511.738464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.965125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8514.585827] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8519.481978] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8523.727608] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8528.465429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8528.667586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.373340] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8538.669845] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8543.456186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8543.642658] Lustre: lustre-OST0001: new disk, initializing [ 8543.646350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8543.654301] Lustre: 382309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8547.630803] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8551.985765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8551.994321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8552.110188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8553.930424] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8561.427052] Lustre: lustre-OST0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 8561.435808] Lustre: Skipped 1 previous similar message [ 8562.161289] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.168417] Lustre: Skipped 2 previous similar messages [ 8566.525042] Lustre: lustre-OST0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 8566.535150] Lustre: Skipped 2 previous similar messages [ 8572.383251] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8572.452160] Lustre: server umount lustre-OST0001 complete [ 8577.508429] LustreError: 381296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.514878] LustreError: 381296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8582.635431] LustreError: 382919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.660110] LustreError: 382919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8587.745292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8587.753295] LustreError: 381295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8587.760450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8587.763389] LustreError: 381295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8587.782069] Lustre: Skipped 6 previous similar messages [ 8592.867125] LustreError: 381294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8592.881372] LustreError: 381294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8593.335218] Lustre: server umount lustre-OST0000 complete [ 8597.985841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8597.995448] Lustre: Skipped 2 previous similar messages [ 8602.432678] Lustre: server umount lustre-MDT0000 complete [ 8603.110031] LustreError: 380204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.119364] LustreError: 380204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8605.268982] LustreError: 379458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488979 with bad export cookie 8957356739374862387 [ 8605.271681] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8605.283878] LustreError: 379458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8611.617256] Lustre: server umount lustre-MDT0001 complete [ 8618.817856] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8620.821990] Key type lgssc unregistered [ 8621.030758] LNet: 384296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8621.034677] LNetError: 384296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8621.050085] LNet: Removed LNI 192.168.202.102@tcp [ 8621.712231] Key type .llcrypt unregistered [ 8621.714224] Key type ._llcrypt unregistered [ 8655.296524] Key type ._llcrypt registered [ 8655.300098] Key type .llcrypt registered [ 8655.375143] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8656.074126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8656.170784] alg: No test for adler32 (adler32-zlib) [ 8657.160440] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8657.356241] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8658.976165] Key type lgssc registered [ 8659.746834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8665.592466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8665.621484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.783537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8666.798495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8666.917722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.786891] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8677.636456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.716622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8677.736076] Lustre: 387073:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8677.944588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8681.935263] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8688.079649] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8692.929760] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8697.314347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.330527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8697.335223] Lustre: Skipped 1 previous similar message [ 8700.873063] Lustre: server umount lustre-MDT0000 complete [ 8703.924439] LustreError: 387064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489078 with bad export cookie 3387136260256700891 [ 8703.938408] LustreError: 387064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8703.939197] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8704.113306] Lustre: server umount lustre-MDT0001 complete [ 8709.623448] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:51:23 (1782489083) [ 8715.727481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.550312] LustreError: 389584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8716.573279] LustreError: 389584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8716.661857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8721.756252] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8721.888378] LustreError: 389585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8727.011526] LustreError: 389584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8732.133406] LustreError: 389585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.280691] LustreError: 389584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8741.632242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8742.373565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8743.002657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8748.422662] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8760.969359] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8770.049876] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8778.979130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8779.289086] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8779.529980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.884334] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8795.748445] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8804.220970] Lustre: Permanently deactivating lustre-OST0000 [ 8804.249543] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8811.063855] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8811.958987] LustreError: 392525:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8811.965965] LustreError: 392525:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8811.970472] LustreError: 392525:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8815.116151] LustreError: 392575:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8815.124887] LustreError: 392575:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8823.509640] Lustre: server umount lustre-OST0000 complete [ 8826.540555] LustreError: 392824:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8826.545160] LustreError: 392824:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8829.920979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8829.931227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.962555] Lustre: Skipped 1 previous similar message [ 8829.976925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8833.504675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8833.507945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8837.844054] Lustre: server umount lustre-MDT0000 complete [ 8838.626271] LustreError: 389581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.644357] LustreError: 389581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8842.472322] LustreError: 389565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489216 with bad export cookie 3387136260256701472 [ 8842.475505] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8842.492528] LustreError: 389565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8851.855700] Lustre: server umount lustre-MDT0001 complete [ 8861.382734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.403545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8869.661426] LustreError: 393985:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8876.993049] Lustre: 394325:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.102@tcp [ 8878.775776] Lustre: server umount MGS complete [ 8892.463350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.241274] LustreError: 395050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.296510] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8893.308474] Lustre: Skipped 1 previous similar message [ 8893.433336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.213145] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8913.513151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.877544] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8913.990077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8919.373497] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8929.966315] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8936.322943] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8938.383581] Lustre: Permanently reactivating lustre-OST0000 [ 8938.396228] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8938.399606] Lustre: Skipped 2 previous similar messages [ 8968.160374] LustreError: 396872:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8976.594086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8977.103217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8984.076549] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8991.880205] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8998.881513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.890099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8998.908542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.913156] Lustre: Skipped 1 previous similar message [ 8999.250705] Lustre: server umount lustre-OST0000 complete [ 9004.002396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9004.017589] Lustre: Skipped 2 previous similar messages [ 9009.063913] Lustre: server umount lustre-MDT0000 complete [ 9009.123266] LustreError: 396193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.140243] LustreError: 396193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9012.917668] LustreError: 395032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489387 with bad export cookie 3387136260256702522 [ 9012.932536] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9012.943356] LustreError: 395032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9013.439854] Lustre: server umount lustre-MDT0001 complete [ 9025.022435] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9027.930068] Key type lgssc unregistered [ 9028.332800] LNet: 399467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9028.343877] LNetError: 399467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9028.358310] LNet: Removed LNI 192.168.202.102@tcp [ 9029.367336] Key type .llcrypt unregistered [ 9029.369411] Key type ._llcrypt unregistered [ 9048.593621] Key type ._llcrypt registered [ 9048.594865] Key type .llcrypt registered [ 9048.701767] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9062.391908] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9063.382162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9063.448853] alg: No test for adler32 (adler32-zlib) [ 9064.601892] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 9065.010986] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9066.783175] Key type lgssc registered [ 9067.887507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9104.888518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9104.919844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.169571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9106.237890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9106.370710] Lustre: lustre-MDT0000: new disk, initializing [ 9106.494180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.516780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9111.215615] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9122.821736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.958371] Lustre: 402615:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9123.034726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9123.041982] Lustre: Skipped 1 previous similar message [ 9123.128542] Lustre: lustre-MDT0001: new disk, initializing [ 9123.236346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9123.284088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9123.293930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9128.950761] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9139.412741] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9145.482886] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9153.139429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.497649] Lustre: lustre-OST0000: new disk, initializing [ 9153.501243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9153.511843] Lustre: 404583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9153.651739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9162.183544] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9162.374552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9162.389928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9162.613668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9172.813941] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9177.842337] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9178.117613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9183.423944] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9183.643605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9188.322430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.329775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.356108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.358659] Lustre: Skipped 1 previous similar message [ 9191.728742] Lustre: server umount lustre-OST0000 complete [ 9198.561831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.581811] Lustre: Skipped 1 previous similar message [ 9198.591359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.597701] Lustre: Skipped 1 previous similar message [ 9201.893959] Lustre: server umount lustre-MDT0000 complete [ 9205.565355] LustreError: 402606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489580 with bad export cookie 1927726097088422995 [ 9205.566358] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.580279] LustreError: 402606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9205.932940] Lustre: server umount lustre-MDT0001 complete [ 9216.112181] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9217.791524] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:59:51 (1782489591) [ 9225.154092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.758031] LustreError: 407044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.779143] LustreError: 407044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9225.871306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.691852] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9231.343296] LustreError: 407045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9236.450095] LustreError: 407044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9240.138869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.601970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9245.951357] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9253.840845] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9259.985710] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9268.697192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.140204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.549678] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9285.364744] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9288.011180] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9291.934138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9300.963877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9300.967206] LustreError: Skipped 1 previous similar message [ 9300.970926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9300.977783] Lustre: Skipped 1 previous similar message [ 9300.996765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9306.017897] Lustre: server umount lustre-OST0000 complete [ 9311.201272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9311.202636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9311.214521] Lustre: Skipped 2 previous similar messages [ 9311.242750] Lustre: Skipped 2 previous similar messages [ 9316.321826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9316.334630] Lustre: Skipped 1 previous similar message [ 9316.948300] Lustre: server umount lustre-MDT0000 complete [ 9320.430097] LustreError: 407026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489694 with bad export cookie 1927726097088423884 [ 9320.436826] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9320.449432] LustreError: 407026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9320.757849] Lustre: server umount lustre-MDT0001 complete [ 9335.347950] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9338.839558] Key type lgssc unregistered [ 9339.308242] LNet: 410781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9339.315869] LNetError: 410781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9339.333755] LNet: Removed LNI 192.168.202.102@tcp [ 9340.339132] Key type .llcrypt unregistered [ 9340.340694] Key type ._llcrypt unregistered [ 9374.254346] Key type ._llcrypt registered [ 9374.255833] Key type .llcrypt registered [ 9374.363580] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9374.567253] Key type .llcrypt unregistered [ 9374.569598] Key type ._llcrypt unregistered [ 9390.076648] Key type ._llcrypt registered [ 9390.078744] Key type .llcrypt registered [ 9390.176149] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9391.030161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9391.273891] alg: No test for adler32 (adler32-zlib) [ 9392.373253] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 9392.604857] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9394.263227] Key type lgssc registered [ 9395.258897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9402.417875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9402.430415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.731038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.192921] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9414.123921] LustreError: 412999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.164493] LustreError: 412999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9417.387690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.706139] LustreError: 412998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.733129] LustreError: 412998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9417.736971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.818623] LustreError: 412999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9422.834474] LustreError: 412999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9424.807941] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9427.939466] LustreError: 412994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9427.976206] LustreError: 412994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9428.980459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9432.064780] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9434.092463] LustreError: 412993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.112385] LustreError: 412993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9438.352529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.755826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9442.742170] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9452.956635] Lustre: server umount lustre-OST0000 complete [ 9458.161030] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9458.173526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.208121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9459.170447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9459.174991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.290781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.298806] Lustre: Skipped 2 previous similar messages [ 9467.539804] Lustre: server umount lustre-MDT0000 complete [ 9469.409304] LustreError: 412999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.441688] LustreError: 412999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9470.367306] Lustre: 412309:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782489828/real 1782489828] req@00000000e8d3c190 x1869075989948032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782489844 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9470.409099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9472.027336] LustreError: 412978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489846 with bad export cookie 13797383710860792313 [ 9472.052261] LustreError: 412978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9472.052911] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.818427] Lustre: server umount lustre-MDT0001 complete [ 9488.051242] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9490.719585] Key type lgssc unregistered [ 9491.119799] LNet: 416148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9491.132234] LNetError: 416148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9491.177263] LNet: Removed LNI 192.168.202.102@tcp [ 9491.985811] Key type .llcrypt unregistered [ 9491.989742] Key type ._llcrypt unregistered [ 9516.088974] Key type ._llcrypt registered [ 9516.092685] Key type .llcrypt registered [ 9516.168233] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9517.348761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9517.392241] alg: No test for adler32 (adler32-zlib) [ 9518.471836] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 9518.685782] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9520.327596] Key type lgssc registered [ 9521.324195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9528.679100] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9528.694873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.227371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9549.343990] LustreError: 417539:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 9576.032293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9582.741534] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9590.238600] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9597.757474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9597.884135] LustreError: 416875:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000042af93af x1869076122044672/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9598.287266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.365845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9602.343446] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9603.560095] LustreError: 417565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9603.572466] LustreError: 417565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9608.678536] LustreError: 417564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.878983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.100450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9613.431372] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9619.822814] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9624.930685] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9634.272274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9634.279272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.286967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9635.808953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9635.822090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9635.848181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9639.627180] Lustre: server umount lustre-OST0000 complete [ 9646.065734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9646.073248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9646.091451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9646.104211] Lustre: Skipped 2 previous similar messages [ 9650.088351] Lustre: server umount lustre-MDT0000 complete [ 9652.201139] LustreError: 418577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.232168] LustreError: 418577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9653.872415] LustreError: 419317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490028 with bad export cookie 1346090297060512360 [ 9653.876975] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9653.884070] LustreError: 419317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9654.185683] Lustre: server umount lustre-MDT0001 complete [ 9665.761813] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9669.251961] Key type lgssc unregistered [ 9669.522522] LNet: 421108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9669.529707] LNetError: 421108:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9669.556359] LNet: Removed LNI 192.168.202.102@tcp [ 9670.427884] Key type .llcrypt unregistered [ 9670.430310] Key type ._llcrypt unregistered [ 9694.968316] Key type ._llcrypt registered [ 9694.971558] Key type .llcrypt registered [ 9695.039764] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9696.157112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9696.313043] alg: No test for adler32 (adler32-zlib) [ 9697.360550] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 9697.666141] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9699.351506] Key type lgssc registered [ 9700.419772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9708.573636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9708.591819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.230993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9714.649168] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9719.792359] LustreError: 422526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9719.804237] LustreError: 422526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9722.317588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.682941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9726.934545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9734.505587] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9739.521462] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9745.837062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9746.234259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.205837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9755.458902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9759.042635] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9763.809499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9763.816682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.836455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9765.855900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9765.857214] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9765.862911] Lustre: Skipped 1 previous similar message [ 9765.889639] Lustre: Skipped 1 previous similar message [ 9770.012021] Lustre: server umount lustre-MDT0000 complete [ 9770.977579] LustreError: 423253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9771.002738] LustreError: 423253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9772.158924] LustreError: 425102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.2@tcp arrived at 1782490146 with bad export cookie 4612520739570717308 [ 9772.171669] LustreError: 425102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9773.753298] LustreError: 422506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490148 with bad export cookie 4612520739570716412 [ 9773.762825] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9773.763725] LustreError: 422506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9774.048546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9774.058988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.078941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9774.087416] Lustre: Skipped 2 previous similar messages [ 9776.096277] LustreError: 422527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.099922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9776.118538] LustreError: 422527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9778.146826] LustreError: 423253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.174707] LustreError: 423253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9779.961505] Lustre: server umount lustre-OST0000 complete [ 9783.264801] LustreError: 422523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9783.283511] LustreError: 422523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9786.715739] Lustre: server umount lustre-MDT0001 complete [ 9797.841826] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9800.897534] Key type lgssc unregistered [ 9801.244677] LNet: 426208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9801.268524] LNetError: 426208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9801.283217] LNet: Removed LNI 192.168.202.102@tcp [ 9802.380835] Key type .llcrypt unregistered [ 9802.383094] Key type ._llcrypt unregistered [ 9825.550935] Key type ._llcrypt registered [ 9825.552071] Key type .llcrypt registered [ 9825.621235] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9826.364552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9826.391058] alg: No test for adler32 (adler32-zlib) [ 9827.411830] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 9827.615233] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9829.279237] Key type lgssc registered [ 9830.490212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9838.410383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9838.422800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9840.016950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.299088] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9850.342334] LustreError: 427605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9850.371788] LustreError: 427605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9852.247570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.861287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9857.294925] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9864.299585] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9869.636646] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9876.722878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.968765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.468063] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9885.187901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9889.818560] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9895.394857] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.408990] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9895.417387] Lustre: Skipped 1 previous similar message [ 9895.434050] Lustre: Skipped 1 previous similar message [ 9897.758903] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 9900.525201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9900.537666] Lustre: Skipped 1 previous similar message [ 9901.022763] Lustre: server umount lustre-MDT0001 complete [ 9902.858196] LustreError: 429094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.646343] LustreError: 427600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.677779] LustreError: 427600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9910.758682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9910.766141] LustreError: 427606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9910.776900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.812649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9915.285147] Lustre: server umount lustre-OST0000 complete [ 9915.873544] LustreError: 428621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.195348] Lustre: server umount lustre-MDT0000 complete [ 9933.136163] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9937.105989] Key type lgssc unregistered [ 9937.418771] LNet: 431290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9937.427153] LNetError: 431290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9937.458085] LNet: Removed LNI 192.168.202.102@tcp [ 9939.000879] Key type .llcrypt unregistered [ 9939.003938] Key type ._llcrypt unregistered [ 9975.262903] Key type ._llcrypt registered [ 9975.266574] Key type .llcrypt registered [ 9975.413947] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9993.854445] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9995.406533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9995.432334] alg: No test for adler32 (adler32-zlib) [ 9996.758558] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 9997.138515] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9998.911368] Key type lgssc registered [10001.432774] Lustre: Echo OBD driver; http://www.lustre.org/ [10045.872161] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10061.304300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10061.356356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10062.695201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10062.731894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10062.877245] Lustre: lustre-MDT0000: new disk, initializing [10062.977045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10063.003585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10067.058734] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10081.729847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10082.016242] Lustre: lustre-OST0000: new disk, initializing [10082.022707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10082.034730] Lustre: Skipped 1 previous similar message [10082.053458] Lustre: 436332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10082.178922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10086.735212] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10092.206345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10092.226651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10092.277250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10095.064982] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10107.839165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.958086] Lustre: 435313:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10108.040844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10108.045072] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (not set up) [10108.136452] Lustre: lustre-MDT0001: new disk, initializing [10108.298026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10108.341244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10108.356907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10113.280371] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10122.456271] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10122.702821] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10127.288345] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10127.548778] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10133.790365] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [10135.523660] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.535190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10135.547908] Lustre: Skipped 1 previous similar message [10137.889394] Lustre: server umount lustre-MDT0001 complete [10145.766410] LustreError: 435324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10145.801725] LustreError: 435324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10150.906803] LustreError: 435319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.942769] LustreError: 435319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10156.000499] LustreError: 435325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10156.002925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.033937] Lustre: Skipped 1 previous similar message [10156.045735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10161.128518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10161.134503] LustreError: 435324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.923306] Lustre: server umount lustre-OST0000 complete [10171.966494] Lustre: server umount lustre-MDT0000 complete [10186.129813] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10188.904501] Key type lgssc unregistered [10189.219671] LNet: 440082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10189.228870] LNetError: 440082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10190.245576] LNet: Removed LNI 192.168.202.102@tcp [10191.107673] Key type .llcrypt unregistered [10191.110453] Key type ._llcrypt unregistered [10229.589870] Key type ._llcrypt registered [10229.592371] Key type .llcrypt registered [10229.693750] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10229.872471] Key type .llcrypt unregistered [10229.873881] Key type ._llcrypt unregistered [10248.359735] Key type ._llcrypt registered [10248.362593] Key type .llcrypt registered [10248.490147] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10250.328337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10250.373627] alg: No test for adler32 (adler32-zlib) [10251.563957] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [10251.900141] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10253.759325] Key type lgssc registered [10254.974974] Lustre: Echo OBD driver; http://www.lustre.org/ [10283.606301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10283.641242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.851943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10284.872485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10284.961661] Lustre: lustre-MDT0000: new disk, initializing [10285.053974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10285.078377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10289.051815] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10299.656478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10299.792434] Lustre: 443460:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10299.839809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10299.843294] Lustre: Skipped 1 previous similar message [10299.896249] Lustre: lustre-MDT0001: new disk, initializing [10299.971991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10300.028773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10300.044152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10304.281871] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10314.172307] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10319.347226] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10326.561169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10326.819298] Lustre: lustre-OST0000: new disk, initializing [10326.822961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10326.827967] Lustre: 445428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10326.912556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10334.638441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10336.333616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10336.348116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10336.466757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10344.714126] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10351.075143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10351.083803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10351.108896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.100622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10352.103522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10352.130521] Lustre: Skipped 1 previous similar message [10357.019459] Lustre: server umount lustre-MDT0000 complete [10361.011846] LustreError: 443450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490735 with bad export cookie 4545060381298082622 [10361.014703] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10361.017756] LustreError: 443450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10361.297332] Lustre: server umount lustre-MDT0001 complete [10375.931114] Lustre: server umount lustre-OST0000 complete [10388.154545] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:19:21 (1782490761) [10398.433715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.016760] LustreError: 447456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10399.053875] LustreError: 447456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10399.144079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10404.330839] LustreError: 447457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.422679] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10409.447714] LustreError: 447456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.417068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10413.989397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10418.859442] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10426.109799] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10433.066869] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10453.417336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10453.562414] Lustre: Found index 0 for lustre-OST0000, updating log [10453.768524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10461.178740] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10469.785179] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10470.194664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10479.594312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10479.609506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10479.654033] Lustre: Skipped 1 previous similar message [10479.674550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10487.351114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10487.357025] Lustre: Skipped 4 previous similar messages [10491.363716] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10491.612754] Lustre: server umount lustre-OST0000 complete [10508.305145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.316069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.323724] Lustre: Skipped 2 previous similar messages [10508.333833] Lustre: Skipped 1 previous similar message [10514.628893] Lustre: server umount lustre-MDT0000 complete [10515.107125] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [10518.499368] LustreError: 447457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10518.541129] LustreError: 447457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10519.631562] LustreError: 447436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490894 with bad export cookie 4545060381298084015 [10519.633299] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10519.640924] LustreError: 447436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10520.159249] Lustre: server umount lustre-MDT0001 complete [10530.024792] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:21:43 (1782490903) [10579.616272] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [11153.943121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.372315] LustreError: 453870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.390526] LustreError: 453870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11154.420335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11157.702832] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11159.522801] LustreError: 453871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.620479] LustreError: 453870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.345272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.746460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11170.405123] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11177.317255] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11182.485950] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11202.475477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.786949] Lustre: Found index 0 for lustre-OST0000, updating log [11203.184717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11206.303366] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [11208.863731] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11216.789506] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11218.947837] Lustre: server umount lustre-OST0000 complete [11224.291159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.295955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11224.311232] Lustre: Skipped 1 previous similar message [11224.322063] Lustre: Skipped 3 previous similar messages [11225.378803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11225.397491] Lustre: Skipped 1 previous similar message [11227.427281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11227.433139] Lustre: Skipped 2 previous similar messages [11229.626622] Lustre: server umount lustre-MDT0000 complete [11230.496749] LustreError: 454873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.523222] LustreError: 454873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11233.126217] LustreError: 453852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491607 with bad export cookie 4545060381298085100 [11233.144154] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11233.146057] LustreError: 453852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11233.414850] Lustre: server umount lustre-MDT0001 complete [11261.423209] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11265.695269] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [11268.931346] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11296.682971] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:34:30 (1782491670) [11308.317392] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11310.323443] Key type lgssc unregistered [11310.537209] LNet: 460474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11310.541974] LNetError: 460474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11417.069937] LNet: Removed LNI 192.168.202.102@tcp [11417.853108] Key type .llcrypt unregistered [11417.859743] Key type ._llcrypt unregistered [11430.357935] Key type ._llcrypt registered [11430.361543] Key type .llcrypt registered [11430.451370] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11431.239204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11431.299230] alg: No test for adler32 (adler32-zlib) [11432.411826] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [11432.601468] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11434.240416] Key type lgssc registered [11435.099594] Lustre: Echo OBD driver; http://www.lustre.org/ [11438.380377] LNet: Added LNI 192.168.202.102@tcp100 [8/256/0/180] [11439.144096] LNet: Added LNI 192.168.202.102@tcp101 [8/256/0/180] [11439.804873] LNet: Added LNI 192.168.202.102@tcp102 [8/256/0/180] [11440.468599] LNet: Added LNI 192.168.202.102@tcp103 [8/256/0/180] [11441.201481] LNet: Added LNI 192.168.202.102@tcp104 [8/256/0/180] [11441.971133] LNet: Added LNI 192.168.202.102@tcp105 [8/256/0/180] [11442.730995] LNet: Added LNI 192.168.202.102@tcp106 [8/256/0/180] [11443.437389] LNet: Added LNI 192.168.202.102@tcp107 [8/256/0/180] [11444.201187] LNet: Added LNI 192.168.202.102@tcp108 [8/256/0/180] [11445.050948] LNet: Added LNI 192.168.202.102@tcp109 [8/256/0/180] [11445.689304] LNet: Added LNI 192.168.202.102@tcp110 [8/256/0/180] [11446.411685] LNet: Added LNI 192.168.202.102@tcp111 [8/256/0/180] [11447.180042] LNet: Added LNI 192.168.202.102@tcp112 [8/256/0/180] [11447.941649] LNet: Added LNI 192.168.202.102@tcp113 [8/256/0/180] [11448.678446] LNet: Added LNI 192.168.202.102@tcp114 [8/256/0/180] [11449.381601] LNet: Added LNI 192.168.202.102@tcp115 [8/256/0/180] [11450.071627] LNet: Added LNI 192.168.202.102@tcp116 [8/256/0/180] [11450.800837] LNet: Added LNI 192.168.202.102@tcp117 [8/256/0/180] [11451.486235] LNet: Added LNI 192.168.202.102@tcp118 [8/256/0/180] [11452.092218] LNet: Added LNI 192.168.202.102@tcp119 [8/256/0/180] [11452.880762] LNet: Added LNI 192.168.202.102@tcp120 [8/256/0/180] [11453.569150] LNet: Added LNI 192.168.202.102@tcp121 [8/256/0/180] [11454.297914] LNet: Added LNI 192.168.202.102@tcp122 [8/256/0/180] [11454.973484] LNet: Added LNI 192.168.202.102@tcp123 [8/256/0/180] [11455.746471] LNet: Added LNI 192.168.202.102@tcp124 [8/256/0/180] [11456.388790] LNet: Added LNI 192.168.202.102@tcp125 [8/256/0/180] [11457.136684] LNet: Added LNI 192.168.202.102@tcp126 [8/256/0/180] [11457.819931] LNet: Added LNI 192.168.202.102@tcp127 [8/256/0/180] [11458.549725] LNet: Added LNI 192.168.202.102@tcp128 [8/256/0/180] [11459.369330] LNet: Added LNI 192.168.202.102@tcp129 [8/256/0/180] [11460.005482] LNet: Added LNI 192.168.202.102@tcp130 [8/256/0/180] [11460.652580] LNet: Added LNI 192.168.202.102@tcp131 [8/256/0/180] [11461.357506] LNet: Added LNI 192.168.202.102@tcp132 [8/256/0/180] [11462.029278] LNet: Added LNI 192.168.202.102@tcp133 [8/256/0/180] [11462.698624] LNet: Added LNI 192.168.202.102@tcp134 [8/256/0/180] [11463.390832] LNet: Added LNI 192.168.202.102@tcp135 [8/256/0/180] [11469.250408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11469.270691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11470.350412] Lustre: 463486:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [11470.420043] LNetError: 463510:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11470.427170] LNetError: Protocol error connecting to 192.168.202.102@tcp121 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11470.495159] LNet: 7 local NIs in recovery (showing 7): 192.168.202.102@tcp121, 192.168.202.102@tcp126, 192.168.202.102@tcp124, 192.168.202.102@tcp125, 192.168.202.102@tcp129, 192.168.202.102@tcp127, 192.168.202.102@tcp128 [11470.552299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11470.576443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11470.676772] Lustre: lustre-MDT0000: new disk, initializing [11470.749888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11470.761780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11471.458029] LNetError: 463517:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11471.472084] LNetError: 463517:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [11471.480921] LNetError: Protocol error connecting to 192.168.202.102@tcp103 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11471.487161] LNetError: Skipped 9 previous similar messages [11473.568338] LNetError: 460943:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11473.582082] LNetError: 460943:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [11473.593864] LNetError: Protocol error connecting to 192.168.202.102@tcp126 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11473.606944] LNetError: Skipped 6 previous similar messages [11473.691272] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11477.665125] LNetError: 463513:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11477.680419] LNetError: 463513:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [11477.692927] LNetError: Protocol error connecting to 192.168.202.102@tcp126 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11481.922868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.038140] Lustre: 463501:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11482.094846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11482.098744] Lustre: Skipped 1 previous similar message [11482.182629] Lustre: lustre-MDT0001: new disk, initializing [11482.217287] Lustre: 464307:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [11482.230490] Lustre: 464307:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11482.316862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11482.355322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11482.368406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11485.032694] Lustre: 464500:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [11485.040823] Lustre: 464500:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11485.869475] LNetError: 463507:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11485.877768] LNetError: Protocol error connecting to 192.168.202.102@tcp126 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11486.327576] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11494.500462] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11498.593861] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11500.193352] LNetError: 463512:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11500.206164] LNetError: Protocol error connecting to 192.168.202.102@tcp103 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11503.263942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11503.464506] Lustre: lustre-OST0000: new disk, initializing [11503.467609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11503.471938] Lustre: 465478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11503.522352] Lustre: 465483:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [11503.527443] Lustre: 465483:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11503.564078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11507.988650] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11512.360453] Lustre: 466047:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.102@tcp: can't add NID 192.168.202.102@tcp130: rc = -75 [11512.365852] Lustre: 466047:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11512.406071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11512.413075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11512.499977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11515.123156] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11516.581248] LNetError: 460943:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11516.588636] LNetError: 460943:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [11516.595973] LNetError: Protocol error connecting to 192.168.202.102@tcp122 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11516.605870] LNetError: Skipped 7 previous similar messages [11523.043151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11523.044096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11523.056180] LustreError: Skipped 1 previous similar message [11523.070280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11526.856530] Lustre: server umount lustre-OST0000 complete [11529.889271] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp126 [11533.281862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11533.305580] Lustre: Skipped 1 previous similar message [11533.314975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11533.324067] Lustre: Skipped 2 previous similar messages [11536.122205] Lustre: server umount lustre-MDT0000 complete [11539.102938] LustreError: 463494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491913 with bad export cookie 11746598979039936008 [11539.103704] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11539.109309] LustreError: 463494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11539.351995] Lustre: server umount lustre-MDT0001 complete [11547.835751] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11550.293569] Key type lgssc unregistered [11550.370738] LNetError: 463517:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.102 [11550.380555] LNetError: 463517:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [11550.389876] LNetError: Protocol error connecting to 192.168.202.102@tcp121 at host 192.168.202.102:988: is it running a compatible version of Lustre? [11550.396307] LNetError: Skipped 14 previous similar messages [11550.628724] LNet: 467388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11550.649331] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11550.662895] LNet: Removed LNI 192.168.202.102@tcp [11551.714778] LNet: Removed LNI 192.168.202.102@tcp100 [11551.733893] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11551.746820] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11551.754680] LNet: Removed LNI 192.168.202.102@tcp101 [11552.799224] LNet: Removed LNI 192.168.202.102@tcp102 [11552.805803] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11552.809015] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11552.835237] LNet: Removed LNI 192.168.202.102@tcp103 [11553.890217] LNet: Removed LNI 192.168.202.102@tcp104 [11554.912710] LNet: Removed LNI 192.168.202.102@tcp105 [11554.915047] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11554.927044] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11554.938588] LNet: Removed LNI 192.168.202.102@tcp106 [11555.999365] LNet: Removed LNI 192.168.202.102@tcp107 [11557.023201] LNet: Removed LNI 192.168.202.102@tcp108 [11558.047515] LNet: Removed LNI 192.168.202.102@tcp109 [11559.071206] LNet: Removed LNI 192.168.202.102@tcp110 [11559.074839] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11559.081504] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11559.086383] LNet: Removed LNI 192.168.202.102@tcp111 [11560.095151] LNet: Removed LNI 192.168.202.102@tcp112 [11561.119670] LNet: Removed LNI 192.168.202.102@tcp113 [11562.143168] LNet: Removed LNI 192.168.202.102@tcp114 [11563.167160] LNet: Removed LNI 192.168.202.102@tcp115 [11564.191130] LNet: Removed LNI 192.168.202.102@tcp116 [11565.215134] LNet: Removed LNI 192.168.202.102@tcp117 [11566.239128] LNet: Removed LNI 192.168.202.102@tcp118 [11567.263190] LNet: Removed LNI 192.168.202.102@tcp119 [11567.268553] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11567.273965] LNetError: 467388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11568.287667] LNet: Removed LNI 192.168.202.102@tcp120 [11569.311355] LNet: Removed LNI 192.168.202.102@tcp121 [11570.335318] LNet: Removed LNI 192.168.202.102@tcp122 [11571.359160] LNet: Removed LNI 192.168.202.102@tcp123 [11572.383145] LNet: Removed LNI 192.168.202.102@tcp124 [11573.407382] LNet: Removed LNI 192.168.202.102@tcp125 [11573.411247] LNet: Removed LNI 192.168.202.102@tcp126 [11574.432332] LNet: Removed LNI 192.168.202.102@tcp127 [11575.456337] LNet: Removed LNI 192.168.202.102@tcp128 [11576.479184] LNet: Removed LNI 192.168.202.102@tcp129 [11576.483393] LNet: Removed LNI 192.168.202.102@tcp130 [11576.489326] LNet: Removed LNI 192.168.202.102@tcp131 [11576.491527] LNet: Removed LNI 192.168.202.102@tcp132 [11576.493857] LNet: Removed LNI 192.168.202.102@tcp133 [11576.497199] LNet: Removed LNI 192.168.202.102@tcp134 [11576.509479] LNet: Removed LNI 192.168.202.102@tcp135 [11577.195260] Key type .llcrypt unregistered [11577.197249] Key type ._llcrypt unregistered [11593.250913] Key type ._llcrypt registered [11593.252578] Key type .llcrypt registered [11593.337448] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11605.356757] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11606.485557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11606.505221] alg: No test for adler32 (adler32-zlib) [11607.538115] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [11607.749336] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11609.375191] Key type lgssc registered [11610.155709] Lustre: Echo OBD driver; http://www.lustre.org/ [11635.325878] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:40:09 (1782492009) [11648.167467] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11650.308725] Key type lgssc unregistered [11650.585124] LNet: 471453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11650.591693] LNetError: 471453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11650.627598] LNet: Removed LNI 192.168.202.102@tcp [11651.276949] Key type .llcrypt unregistered [11651.278926] Key type ._llcrypt unregistered [11663.575663] Key type ._llcrypt registered [11663.577865] Key type .llcrypt registered [11663.686200] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11664.394168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11664.453077] alg: No test for adler32 (adler32-zlib) [11665.456332] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [11665.636737] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11667.295185] Key type lgssc registered [11668.147514] Lustre: Echo OBD driver; http://www.lustre.org/ [11675.722902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11675.750259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.989970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11677.020511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11677.113159] Lustre: lustre-MDT0000: new disk, initializing [11677.197105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11677.219329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11680.739666] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11688.501668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11688.588156] Lustre: 472644:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11688.652617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11688.657281] Lustre: Skipped 1 previous similar message [11688.748475] Lustre: lustre-MDT0001: new disk, initializing [11688.835919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11688.883708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11688.892079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11692.430242] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11699.355362] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11703.536219] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11708.386268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.602930] Lustre: lustre-OST0000: new disk, initializing [11708.609980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11708.617462] Lustre: 474608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11708.705490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11713.275413] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11715.185749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11715.194228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11715.277781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11720.888724] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11723.769202] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing lnet_if_list [11727.789460] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11727.813690] LNet: Added LNI 192.168.202.102@tcp10 [8/256/0/180] [11732.989283] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11733.012780] LNet: Added LNI 192.168.202.102@tcp11 [8/256/0/180] [11736.120359] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11736.146221] LNet: Added LNI 192.168.202.102@tcp12 [8/256/0/180] [11739.659045] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11739.691638] LNet: Added LNI 192.168.202.102@tcp13 [8/256/0/180] [11742.643286] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11742.673178] LNet: Added LNI 192.168.202.102@tcp14 [8/256/0/180] [11746.055450] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11746.104325] LNet: Added LNI 192.168.202.102@tcp15 [8/256/0/180] [11749.427790] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11749.451362] LNet: Added LNI 192.168.202.102@tcp16 [8/256/0/180] [11752.808853] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11752.830686] LNet: Added LNI 192.168.202.102@tcp17 [8/256/0/180] [11756.265521] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11756.295352] LNet: Added LNI 192.168.202.102@tcp18 [8/256/0/180] [11760.641917] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11760.669077] LNet: Added LNI 192.168.202.102@tcp19 [8/256/0/180] [11764.682399] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11764.707304] LNet: Added LNI 192.168.202.102@tcp20 [8/256/0/180] [11768.138561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11768.165588] LNet: Added LNI 192.168.202.102@tcp21 [8/256/0/180] [11771.695811] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11771.714424] LNet: Added LNI 192.168.202.102@tcp22 [8/256/0/180] [11775.574268] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11775.593558] LNet: Added LNI 192.168.202.102@tcp23 [8/256/0/180] [11778.807890] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11778.824395] LNet: Added LNI 192.168.202.102@tcp24 [8/256/0/180] [11782.209585] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11782.224378] LNet: Added LNI 192.168.202.102@tcp25 [8/256/0/180] [11785.954798] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11785.971457] LNet: Added LNI 192.168.202.102@tcp26 [8/256/0/180] [11790.118723] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11790.135074] LNet: Added LNI 192.168.202.102@tcp27 [8/256/0/180] [11794.259875] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11794.296601] LNet: Added LNI 192.168.202.102@tcp28 [8/256/0/180] [11799.389314] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11799.423875] LNet: Added LNI 192.168.202.102@tcp29 [8/256/0/180] [11803.608535] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11803.634721] LNet: Added LNI 192.168.202.102@tcp30 [8/256/0/180] [11807.717525] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11807.738877] LNet: Added LNI 192.168.202.102@tcp31 [8/256/0/180] [11811.776697] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11811.795444] LNet: Added LNI 192.168.202.102@tcp32 [8/256/0/180] [11816.236794] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11816.260286] LNet: Added LNI 192.168.202.102@tcp33 [8/256/0/180] [11820.666274] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11820.688531] LNet: Added LNI 192.168.202.102@tcp34 [8/256/0/180] [11825.083805] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11825.105634] LNet: Added LNI 192.168.202.102@tcp35 [8/256/0/180] [11829.544565] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11829.563760] LNet: Added LNI 192.168.202.102@tcp36 [8/256/0/180] [11833.908370] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11833.935758] LNet: Added LNI 192.168.202.102@tcp37 [8/256/0/180] [11838.890686] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11838.908746] LNet: Added LNI 192.168.202.102@tcp38 [8/256/0/180] [11843.874708] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11843.895351] LNet: Added LNI 192.168.202.102@tcp39 [8/256/0/180] [11849.023549] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11849.046928] LNet: Added LNI 192.168.202.102@tcp40 [8/256/0/180] [11854.844849] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11854.862648] LNet: Added LNI 192.168.202.102@tcp41 [8/256/0/180] [11860.066654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11860.086705] LNet: Added LNI 192.168.202.102@tcp42 [8/256/0/180] [11865.143436] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11865.164761] LNet: Added LNI 192.168.202.102@tcp43 [8/256/0/180] [11870.607441] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11870.627498] LNet: Added LNI 192.168.202.102@tcp44 [8/256/0/180] [11876.148855] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11876.175708] LNet: Added LNI 192.168.202.102@tcp45 [8/256/0/180] [11881.900343] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11881.915159] LNet: Added LNI 192.168.202.102@tcp46 [8/256/0/180] [11887.634884] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11887.658338] LNet: Added LNI 192.168.202.102@tcp47 [8/256/0/180] [11892.826042] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11892.860628] LNet: Added LNI 192.168.202.102@tcp48 [8/256/0/180] [11898.365834] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11898.391130] LNet: Added LNI 192.168.202.102@tcp49 [8/256/0/180] [11904.355886] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11904.382525] LNet: Added LNI 192.168.202.102@tcp50 [8/256/0/180] [11919.328561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11919.335842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.350961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11920.362913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11920.363929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11920.380937] Lustre: Skipped 2 previous similar messages [11923.800296] Lustre: server umount lustre-MDT0000 complete [11926.284421] LustreError: 472636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492300 with bad export cookie 17314292141863074556 [11926.286501] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11926.299954] LustreError: 472636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11926.509402] Lustre: server umount lustre-MDT0001 complete [11938.578309] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11944.704863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11946.976199] Lustre: 471917:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782492305/real 1782492305] req@0000000031815987 x1869078373465216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782492321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11946.997224] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11947.002720] Lustre: Skipped 1 previous similar message [11947.999520] Lustre: 471918:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782492306/real 1782492306] req@0000000049cb5255 x1869078373465344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782492322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11951.135581] LustreError: 471915:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000013f3f438 x1869078373466112/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11951.351386] LustreError: 474602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11951.382435] LustreError: 474602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11951.423130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11952.481512] Lustre: 471919:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782492310/real 1782492310] req@00000000ab5ff02a x1869078373465472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782492326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11952.483899] LustreError: 474603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.430355] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11956.521444] LustreError: 474602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.560502] Lustre: 471916:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782492315/real 1782492315] req@00000000dce774e5 x1869078373465728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782492331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11960.289685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11960.557152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11963.490591] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11965.935542] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11968.824858] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11973.155933] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11976.167598] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11986.411225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.422715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11986.422995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11986.434566] Lustre: Skipped 1 previous similar message [11987.528895] Lustre: server umount lustre-OST0000 complete [11991.520895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11991.522048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.528643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.537038] Lustre: Skipped 2 previous similar messages [11991.541986] Lustre: Skipped 1 previous similar message [11996.640913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11996.647425] Lustre: Skipped 1 previous similar message [12004.320327] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12004.488425] Lustre: server umount lustre-MDT0000 complete [12006.883702] LustreError: 499969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12006.902173] LustreError: 499969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12007.005832] LustreError: 472636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492381 with bad export cookie 17314292141863076950 [12007.009132] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12007.012939] LustreError: 472636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12007.253744] Lustre: server umount lustre-MDT0001 complete [12015.894375] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12017.611505] Key type lgssc unregistered [12017.820964] LNet: 501593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12017.826692] LNetError: 501593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12017.831449] LNet: Removed LNI 192.168.202.102@tcp [12017.833558] LNet: Removed LNI 192.168.202.102@tcp10 [12017.836195] LNet: Removed LNI 192.168.202.102@tcp11 [12017.838531] LNet: Removed LNI 192.168.202.102@tcp12 [12017.841230] LNet: Removed LNI 192.168.202.102@tcp13 [12017.844216] LNet: Removed LNI 192.168.202.102@tcp14 [12017.846579] LNet: Removed LNI 192.168.202.102@tcp15 [12017.848967] LNet: Removed LNI 192.168.202.102@tcp16 [12017.850642] LNet: Removed LNI 192.168.202.102@tcp17 [12017.853441] LNet: Removed LNI 192.168.202.102@tcp18 [12017.858071] LNet: Removed LNI 192.168.202.102@tcp19 [12017.861649] LNet: Removed LNI 192.168.202.102@tcp20 [12017.864912] LNet: Removed LNI 192.168.202.102@tcp21 [12017.870539] LNet: Removed LNI 192.168.202.102@tcp22 [12017.872803] LNet: Removed LNI 192.168.202.102@tcp23 [12017.875367] LNet: Removed LNI 192.168.202.102@tcp24 [12017.878212] LNet: Removed LNI 192.168.202.102@tcp25 [12017.881340] LNet: Removed LNI 192.168.202.102@tcp26 [12017.883663] LNet: Removed LNI 192.168.202.102@tcp27 [12017.890992] LNet: Removed LNI 192.168.202.102@tcp28 [12017.894571] LNet: Removed LNI 192.168.202.102@tcp29 [12017.897592] LNet: Removed LNI 192.168.202.102@tcp30 [12017.900689] LNet: Removed LNI 192.168.202.102@tcp31 [12017.903229] LNet: Removed LNI 192.168.202.102@tcp32 [12017.905827] LNet: Removed LNI 192.168.202.102@tcp33 [12017.908421] LNet: Removed LNI 192.168.202.102@tcp34 [12017.911270] LNet: Removed LNI 192.168.202.102@tcp35 [12017.913914] LNet: Removed LNI 192.168.202.102@tcp36 [12017.915883] LNet: Removed LNI 192.168.202.102@tcp37 [12017.918299] LNet: Removed LNI 192.168.202.102@tcp38 [12017.920574] LNet: Removed LNI 192.168.202.102@tcp39 [12017.923287] LNet: Removed LNI 192.168.202.102@tcp40 [12017.926873] LNet: Removed LNI 192.168.202.102@tcp41 [12017.929287] LNet: Removed LNI 192.168.202.102@tcp42 [12017.931120] LNet: Removed LNI 192.168.202.102@tcp43 [12017.933613] LNet: Removed LNI 192.168.202.102@tcp44 [12017.935534] LNet: Removed LNI 192.168.202.102@tcp45 [12017.937768] LNet: Removed LNI 192.168.202.102@tcp46 [12017.940253] LNet: Removed LNI 192.168.202.102@tcp47 [12017.943333] LNet: Removed LNI 192.168.202.102@tcp48 [12017.945659] LNet: Removed LNI 192.168.202.102@tcp49 [12017.958200] LNet: Removed LNI 192.168.202.102@tcp50 [12018.529434] Key type .llcrypt unregistered [12018.534237] Key type ._llcrypt unregistered [12032.377403] Key type ._llcrypt registered [12032.379477] Key type .llcrypt registered [12032.457563] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12042.502518] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12043.345281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12043.374122] alg: No test for adler32 (adler32-zlib) [12044.424962] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [12044.614555] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12046.239356] Key type lgssc registered [12047.007927] Lustre: Echo OBD driver; http://www.lustre.org/ [12069.799831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12069.835432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.117786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12071.182049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12071.235602] Lustre: lustre-MDT0000: new disk, initializing [12071.357583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12071.376633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12075.162943] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12090.731316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.942633] Lustre: 504766:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12091.003743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12091.006988] Lustre: Skipped 1 previous similar message [12091.085373] Lustre: lustre-MDT0001: new disk, initializing [12091.156388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.187184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12091.201329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12103.688283] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12120.701865] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12130.013405] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12140.952169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12141.404807] Lustre: *** cfs_fail_loc=90f, val=0*** [12141.566751] Lustre: lustre-OST0000: new disk, initializing [12141.570879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12141.583342] Lustre: 506786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12141.894345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12150.681045] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12151.428590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12151.441299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12151.590810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12163.324602] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12172.263924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12172.277195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12172.301215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12172.308880] Lustre: Skipped 1 previous similar message [12177.225957] Lustre: server umount lustre-OST0000 complete [12182.497626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12182.499815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12182.531221] Lustre: Skipped 2 previous similar messages [12182.562037] Lustre: Skipped 1 previous similar message [12187.616508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12187.623367] Lustre: Skipped 1 previous similar message [12191.712800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12191.722021] Lustre: Skipped 1 previous similar message [12196.320730] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12196.708721] Lustre: server umount lustre-MDT0000 complete [12201.037797] LustreError: 504758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492575 with bad export cookie 4924875321894000726 [12201.054504] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12201.688389] Lustre: server umount lustre-MDT0001 complete [12214.663657] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12217.755959] Key type lgssc unregistered [12217.979788] LNet: 508693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12217.985789] LNetError: 508693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12218.014351] LNet: Removed LNI 192.168.202.102@tcp [12218.903497] Key type .llcrypt unregistered [12218.906551] Key type ._llcrypt unregistered [12239.304297] Key type ._llcrypt registered [12239.308470] Key type .llcrypt registered [12239.449799] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12241.331195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12241.342693] alg: No test for adler32 (adler32-zlib) [12242.549131] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [12242.804366] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12244.472483] Key type lgssc registered [12245.730534] Lustre: Echo OBD driver; http://www.lustre.org/ [12253.944115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12253.980854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.413285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12260.568161] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12264.931664] LustreError: 509966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.952057] LustreError: 509966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12269.235362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.656569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.139883] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12285.978552] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12294.081222] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12301.858317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12302.485025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12309.194600] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12316.706764] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12339.013566] loop0: detected capacity change from 0 to 204800000 [12339.250996] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12339.267072] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12339.297862] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12345.787663] loop0: detected capacity change from 0 to 204800000 [12346.111033] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12346.128559] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12346.166320] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12351.804919] loop0: detected capacity change from 0 to 204800000 [12357.323079] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12357.541881] Lustre: fs15246-OST0000: new disk, initializing [12357.545631] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12357.558673] Lustre: 513583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12357.614897] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12364.146964] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12374.724454] loop1: detected capacity change from 0 to 204800000 [12381.344438] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12381.494911] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12381.557648] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12381.616384] Lustre: fs15246-MDT0000: new disk, initializing [12381.802648] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12381.865231] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12382.039232] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12383.077415] Lustre: 514704:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000147630a4 x1869078978413056/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782492773 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12387.646800] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12388.149189] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12388.157546] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12388.229283] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12412.396299] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.413705] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12415.598481] Lustre: server umount fs15246-MDT0000 complete [12417.519130] LustreError: 509961:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.631576] LustreError: 512455:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12426.204538] Lustre: server umount fs15246-OST0000 complete [12443.107077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12443.110280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.121810] LustreError: Skipped 1 previous similar message [12443.142405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.226712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.238177] Lustre: Skipped 2 previous similar messages [12453.346320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12453.351190] Lustre: Skipped 1 previous similar message [12454.368500] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12454.734989] Lustre: server umount lustre-OST0000 complete [12458.474267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12458.482680] Lustre: Skipped 1 previous similar message [12458.487730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12458.497752] Lustre: Skipped 1 previous similar message [12464.666092] Lustre: server umount lustre-MDT0000 complete [12468.714681] LustreError: 511467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.739882] LustreError: 511467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12468.853406] LustreError: 509945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492843 with bad export cookie 17369040377741799243 [12468.855763] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12468.865934] LustreError: 509945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12469.368439] Lustre: server umount lustre-MDT0001 complete [12482.337550] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12485.776683] Key type lgssc unregistered [12486.226925] LNet: 517802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12486.244418] LNetError: 517802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12486.283761] LNet: Removed LNI 192.168.202.102@tcp [12487.617899] Key type .llcrypt unregistered [12487.626093] Key type ._llcrypt unregistered [12509.016346] Key type ._llcrypt registered [12509.023906] Key type .llcrypt registered [12509.181981] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12510.665820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12510.694488] alg: No test for adler32 (adler32-zlib) [12511.803731] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [12512.081940] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12513.783183] Key type lgssc registered [12515.024095] Lustre: Echo OBD driver; http://www.lustre.org/ [12558.785464] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12571.448721] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12610.087901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12610.118477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.429134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12611.473935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12611.568979] Lustre: lustre-MDT0000: new disk, initializing [12611.656904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12611.680392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12615.984653] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12626.282842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.473713] Lustre: 522273:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12626.578164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12626.587468] Lustre: Skipped 1 previous similar message [12626.708611] Lustre: lustre-MDT0001: new disk, initializing [12626.810762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12626.852215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12626.866720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12631.860912] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12642.100858] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12647.732991] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12656.781265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12657.392872] Lustre: lustre-OST0000: new disk, initializing [12657.397809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12657.405445] Lustre: 524241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12657.559408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12664.773974] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12666.967659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12666.975465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12667.121531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12676.172210] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12682.041935] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12682.393401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12689.103094] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12689.338496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12692.967041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12692.967914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.989128] LustreError: Skipped 1 previous similar message [12692.997385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12693.039305] Lustre: Skipped 1 previous similar message [12697.641276] Lustre: server umount lustre-OST0000 complete [12703.200887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12703.210597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12703.210903] Lustre: Skipped 1 previous similar message [12703.240689] Lustre: Skipped 2 previous similar messages [12708.234954] Lustre: server umount lustre-MDT0000 complete [12712.119760] LustreError: 524240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493086 with bad export cookie 11360541598127054331 [12712.129136] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12712.137274] LustreError: 524240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12712.430673] Lustre: server umount lustre-MDT0001 complete [12739.926528] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12752.652671] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12786.544696] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12467 sec ========= 12:59:19 (1782493159) [12788.311964] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:59:21 (1782493161) === [12791.688712] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:59:24 (1782493164) === [12837.303823] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12840.349445] Key type lgssc unregistered [12840.605626] LNet: 530888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12840.609965] LNetError: 530888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12840.627879] LNet: Removed LNI 192.168.202.102@tcp [12841.487599] Key type .llcrypt unregistered [12841.493616] Key type ._llcrypt unregistered