[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 537790242 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002336] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006880] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010042] Yama: becoming mindful. [ 0.011025] SELinux: Initializing. [ 0.012051] *** VALIDATE selinux *** [ 0.020362] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025413] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027062] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] *** VALIDATE tmpfs *** [ 0.031379] *** VALIDATE proc *** [ 0.032192] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034226] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036050] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038024] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.043092] debug: unmapping init [mem 0xffffffff8a459000-0xffffffff8a460fff] [ 0.045972] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047019] ... version: 2 [ 0.048009] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052010] ... fixed-purpose events: 3 [ 0.053008] ... event mask: 000000070000000f [ 0.055194] rcu: Hierarchical SRCU implementation. [ 0.057415] smp: Bringing up secondary CPUs ... [ 0.058527] x86: Booting SMP configuration: [ 0.059015] .... node #0, CPUs: #1 #2 #3 [ 0.067085] smp: Brought up 1 node, 4 CPUs [ 0.069009] smpboot: Max logical packages: 1 [ 0.070008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.095052] node 0 deferred pages initialised in 23ms [ 0.098492] devtmpfs: initialized [ 0.099243] x86/mm: Memory block size: 128MB [ 0.101885] gcov: version magic: 0x41383552 [ 0.103160] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.104063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.105199] pinctrl core: initialized pinctrl subsystem [ 0.106280] [ 0.106694] ************************************************************* [ 0.107011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.108011] ** ** [ 0.109009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.110010] ** ** [ 0.111013] ** This means that this kernel is built to expose internal ** [ 0.112011] ** IOMMU data structures, which may compromise security on ** [ 0.113011] ** your system. ** [ 0.114009] ** ** [ 0.115012] ** If you see this message and you are not debugging the ** [ 0.116011] ** kernel, report this immediately to your vendor! ** [ 0.117011] ** ** [ 0.118008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.119011] ************************************************************* [ 0.120688] NET: Registered protocol family 16 [ 0.121467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.122056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.123054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.124640] cpuidle: using governor menu [ 0.125507] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.128707] PCI: Using configuration type 1 for base access [ 0.131122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.139188] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.141015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.145301] cryptd: max_cpu_qlen set to 1000 [ 0.147072] ACPI: Added _OSI(Module Device) [ 0.149010] ACPI: Added _OSI(Processor Device) [ 0.150011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.152012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.158688] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.166565] ACPI: Interpreter enabled [ 0.167050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.168009] ACPI: Using IOAPIC for interrupt routing [ 0.169077] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.170809] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.182250] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.183034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.184083] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.185077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.187416] acpiphp: Slot [2] registered [ 0.188111] acpiphp: Slot [5] registered [ 0.189155] acpiphp: Slot [6] registered [ 0.190083] acpiphp: Slot [7] registered [ 0.191083] acpiphp: Slot [8] registered [ 0.192083] acpiphp: Slot [9] registered [ 0.193079] acpiphp: Slot [10] registered [ 0.194150] acpiphp: Slot [3] registered [ 0.195073] acpiphp: Slot [4] registered [ 0.196063] acpiphp: Slot [11] registered [ 0.197045] acpiphp: Slot [12] registered [ 0.197916] acpiphp: Slot [13] registered [ 0.198044] acpiphp: Slot [14] registered [ 0.199043] acpiphp: Slot [15] registered [ 0.199940] acpiphp: Slot [16] registered [ 0.200074] acpiphp: Slot [17] registered [ 0.200943] acpiphp: Slot [18] registered [ 0.201047] acpiphp: Slot [19] registered [ 0.201944] acpiphp: Slot [20] registered [ 0.202078] acpiphp: Slot [21] registered [ 0.203044] acpiphp: Slot [22] registered [ 0.203982] acpiphp: Slot [23] registered [ 0.204062] acpiphp: Slot [24] registered [ 0.205064] acpiphp: Slot [25] registered [ 0.206071] acpiphp: Slot [26] registered [ 0.207049] acpiphp: Slot [27] registered [ 0.208055] acpiphp: Slot [28] registered [ 0.209045] acpiphp: Slot [29] registered [ 0.209980] acpiphp: Slot [30] registered [ 0.210047] acpiphp: Slot [31] registered [ 0.210819] PCI host bridge to bus 0000:00 [ 0.211036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.212008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.214018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.217020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.219017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.221014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.223283] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.227283] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.230527] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.240762] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.245048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.247012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.249009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.251010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.253507] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.255746] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.259036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.261669] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.266010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.277022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.283011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.288271] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.296013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.304025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.322015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.331718] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.339016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.348017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.365013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.375322] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.381012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.385013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.397015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.403000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.407012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.414013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.426013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.438886] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.451014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.460017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.476015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.487387] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.493014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.499013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.513016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.527032] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.531404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.534277] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.538384] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.557225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.562097] iommu: Default domain type: Passthrough [ 0.563000] SCSI subsystem initialized [ 0.563000] ACPI: bus type USB registered [ 0.564157] usbcore: registered new interface driver usbfs [ 0.566075] usbcore: registered new interface driver hub [ 0.567159] usbcore: registered new device driver usb [ 0.569132] pps_core: LinuxPPS API ver. 1 registered [ 0.571008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.574050] PTP clock support registered [ 0.576119] EDAC MC: Ver: 3.0.0 [ 0.577856] PCI: Using ACPI for IRQ routing [ 0.579076] NetLabel: Initializing [ 0.580006] NetLabel: domain hash size = 128 [ 0.582009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.584063] NetLabel: unlabeled traffic allowed by default [ 0.585364] vgaarb: loaded [ 0.587620] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.589007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.594259] clocksource: Switched to clocksource kvm-clock [ 0.707195] VFS: Disk quotas dquot_6.6.0 [ 0.708547] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.710100] *** VALIDATE ramfs *** [ 0.710778] *** VALIDATE hugetlbfs *** [ 0.711905] pnp: PnP ACPI init [ 0.713753] pnp: PnP ACPI: found 6 devices [ 0.726809] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.729937] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.731820] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.733511] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.734987] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.737088] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.739727] NET: Registered protocol family 2 [ 0.741265] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.744686] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.747367] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.751410] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.754554] TCP: Hash tables configured (established 65536 bind 65536) [ 0.756755] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.759311] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.761740] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.764413] NET: Registered protocol family 1 [ 0.766798] RPC: Registered named UNIX socket transport module. [ 0.769105] RPC: Registered udp transport module. [ 0.770374] RPC: Registered tcp transport module. [ 0.772178] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.774129] NET: Registered protocol family 44 [ 0.776675] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.778697] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.781135] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.783294] PCI: CLS 0 bytes, default 64 [ 0.784545] Unpacking initramfs... [ 3.218777] debug: unmapping init [mem 0xffff9a63fcc54000-0xffff9a63fffbffff] [ 3.238883] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.242702] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.250264] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.887927] Initialise system trusted keyrings [ 3.889660] Key type blacklist registered [ 3.891369] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.901942] zbud: loaded [ 3.905737] *** VALIDATE nfs *** [ 3.907793] *** VALIDATE nfs4 *** [ 3.910570] pstore: using deflate compression [ 3.916255] Platform Keyring initialized [ 4.025708] NET: Registered protocol family 38 [ 4.027301] Key type asymmetric registered [ 4.028861] Asymmetric key parser 'x509' registered [ 4.031147] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.034419] io scheduler mq-deadline registered [ 4.036320] io scheduler kyber registered [ 4.037584] io scheduler bfq registered [ 4.039312] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.042384] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.044949] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.047951] ACPI: Power Button [PWRF] [ 4.054517] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.062610] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.079939] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.088943] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.102911] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.129141] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.162357] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.166491] Non-volatile memory driver v1.3 [ 4.168100] Linux agpgart interface v0.103 [ 4.194221] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 4.196429] vda: detected capacity change from 0 to 75079680 [ 4.208732] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.211083] vdb: detected capacity change from 0 to 1073741824 [ 4.223426] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.225640] vdc: detected capacity change from 0 to 2621440000 [ 4.245607] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.248098] vdd: detected capacity change from 0 to 2621440000 [ 4.261207] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.264188] vde: detected capacity change from 0 to 4294967296 [ 4.281122] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.284140] vdf: detected capacity change from 0 to 4294967296 [ 4.291130] libphy: Fixed MDIO Bus: probed [ 4.297114] usbcore: registered new interface driver usbserial_generic [ 4.299598] usbserial: USB Serial support registered for generic [ 4.301877] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.306252] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.307650] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.309773] mousedev: PS/2 mouse device common for all mice [ 4.312855] rtc_cmos 00:05: RTC can wake from S4 [ 4.314876] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.318287] rtc_cmos 00:05: registered as rtc0 [ 4.322714] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.326402] intel_pstate: CPU model not supported [ 4.329829] hid: raw HID events driver (C) Jiri Kosina [ 4.331391] usbcore: registered new interface driver usbhid [ 4.333197] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.334968] usbhid: USB HID core driver [ 4.335117] drop_monitor: Initializing network drop monitor service [ 4.335238] Initializing XFRM netlink socket [ 4.336643] NET: Registered protocol family 10 [ 4.345577] Segment Routing with IPv6 [ 4.346680] NET: Registered protocol family 17 [ 4.348296] mpls_gso: MPLS GSO support [ 4.352089] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.355764] RAS: Correctable Errors collector initialized. [ 4.357377] AVX version of gcm_enc/dec engaged. [ 4.358666] AES CTR mode by8 optimization enabled [ 4.434048] sched_clock: Marking stable (4434026434, 0)->(5467570113, -1033543679) [ 4.437143] registered taskstats version 1 [ 4.438546] Loading compiled-in X.509 certificates [ 4.440245] zswap: loaded using pool lzo/zbud [ 4.461200] Key type big_key registered [ 4.472847] Key type encrypted registered [ 4.474113] ima: No TPM chip found, activating TPM-bypass! [ 4.475758] ima: Allocated hash algorithm: sha1 [ 4.477075] ima: No architecture policies found [ 4.478401] evm: Initialising EVM extended attributes: [ 4.479842] evm: security.selinux [ 4.480757] evm: security.ima [ 4.481722] evm: security.capability [ 4.483228] evm: HMAC attrs: 0x1 [ 4.485455] rtc_cmos 00:05: setting system clock to 2026-08-27 14:21:10 UTC (1787840470) [ 4.490852] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 4.493686] debug: unmapping init [mem 0xffffffff8a182000-0xffffffff8a458fff] [ 4.502093] Write protecting the kernel read-only data: 28672k [ 4.505230] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 4.507458] debug: unmapping init [mem 0xffffffff89114000-0xffffffff891fffff] [ 4.539557] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.546358] systemd[1]: Detected virtualization kvm. [ 4.547994] systemd[1]: Detected architecture x86-64. [ 4.549506] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.577954] systemd[1]: No hostname configured. [ 4.579290] systemd[1]: Set hostname to . [ 4.580793] random: systemd: uninitialized urandom read (16 bytes read) [ 4.582929] systemd[1]: Initializing machine ID from random generator. [ 4.941434] random: systemd: uninitialized urandom read (16 bytes read) [ 4.946627] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.958796] random: systemd: uninitialized urandom read (16 bytes read) [ 4.961576] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.018509] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.541209] device-mapper: uevent: version 1.0.3 [ 6.552115] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 8.763091] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.926319] virtio_net virtio0 ens2: renamed from eth0 [ 9.888407] scsi host0: ata_piix [ 9.955822] scsi host1: ata_piix [ 9.960634] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.982087] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.006031] random: crng init done [ 15.007242] random: 7 urandom warning(s) missed due to ratelimiting [ 16.739647] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.348933] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.832849] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.613144] SELinux: Disabled at runtime. [ 23.744449] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.757431] systemd[1]: Detected virtualization kvm. [ 23.759883] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.783994] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.838247] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.940853] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.961256] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.984757] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.033295] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.051321] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 28.446214] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 28.823872] systemd[1]: sysroot-var-lib-nfs-rpc_pipefs.mount: Succeeded. [ 28.852370] systemd[1]: sysroot.mount: Succeeded. [ 28.861388] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 30.780114] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.127274] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.322318] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.793901] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.913235] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit)[ 42.569731] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit)[ 43.767890] NFS: Registering the id_resolver key type [ 43.770302] Key type id_resolver registered [ 43.771913] Key type id_legacy registered [** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [* ] A start job is running for Rebuild …amic Linker Cache (23s / no limit) [** ] A start job is running for Rebuild …amic Linker Cache (24s / no limit) [*** ] A start job is running for Rebuild …amic Linker Cache (24s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-server login: [ 62.942586] hrtimer: interrupt took 10003837 ns [ 145.260510] libcfs: loading out-of-tree module taints kernel. [ 145.355559] Key type ._llcrypt registered [ 145.358796] Key type .llcrypt registered [ 145.498224] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 164.446328] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 165.872391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 165.887264] alg: No test for adler32 (adler32-zlib) [ 167.582971] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 168.439529] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 170.335222] Key type lgssc registered [ 172.397190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 188.651255] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 227.110978] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 240.711388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 240.754282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.051355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 242.096995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 242.229251] Lustre: lustre-MDT0000: new disk, initializing [ 242.337968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.401388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 246.815869] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.492815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 259.619070] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 259.656319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 259.667964] Lustre: Skipped 1 previous similar message [ 259.741411] Lustre: lustre-MDT0001: new disk, initializing [ 259.789372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.815724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 259.825087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 263.637238] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.511470] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 277.013969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.218240] Lustre: lustre-OST0000: new disk, initializing [ 277.221508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 277.226939] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.281408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 280.878382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 280.889526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 280.935767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 282.721314] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.248720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.427345] Lustre: lustre-OST0001: new disk, initializing [ 296.432462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 296.437712] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 296.510678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 301.795694] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.132199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 303.139414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 308.228858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.098732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.611890] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.008564] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing check_logdir /tmp/testlogs/ [ 329.051635] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing yml_node [ 334.074256] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 337.990627] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 341.265959] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 343.623336] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:26:47 EDT 2026 [ 364.200400] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 365.704458] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 384.993568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.999522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.023419] Lustre: Skipped 1 previous similar message [ 385.043015] Lustre: Skipped 3 previous similar messages [ 390.113555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.125821] Lustre: Skipped 3 previous similar messages [ 395.245233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 395.260351] Lustre: Skipped 3 previous similar messages [ 398.304056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.601718] Lustre: server umount lustre-MDT0000 complete [ 403.177703] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840869 with bad export cookie 6379154825143337821 [ 403.193618] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 405.472539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.477980] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.480059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 405.499154] Lustre: Skipped 3 previous similar messages [ 405.549428] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 408.496709] LustreError: 11533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.522637] LustreError: 11533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 409.752449] Lustre: server umount lustre-MDT0001 complete [ 413.720561] Lustre: server umount lustre-OST0000 complete [ 418.032637] Lustre: server umount lustre-OST0001 complete [ 423.981754] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 432.998377] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 474.104258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.512759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 474.602522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 474.704511] Lustre: lustre-MDT0000: new disk, initializing [ 474.762028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.801982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 480.461859] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 492.267962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.448084] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 492.515346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 492.522453] Lustre: Skipped 1 previous similar message [ 492.681545] Lustre: lustre-MDT0001: new disk, initializing [ 492.845366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.923979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 492.948313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 498.896332] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 509.147508] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 515.202520] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 522.428347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.773354] Lustre: lustre-OST0000: new disk, initializing [ 522.776840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 522.781604] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 522.873845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.434823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 524.473046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 524.633365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 530.444548] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 542.146396] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.489784] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 547.805344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 552.846402] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 553.456751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 555.488428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 555.490707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.501821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 555.501829] Lustre: Skipped 3 previous similar messages [ 555.510913] Lustre: Skipped 1 previous similar message [ 565.730648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.745505] Lustre: Skipped 4 previous similar messages [ 569.311132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 569.860673] Lustre: server umount lustre-OST0000 complete [ 580.079328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.088761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.326702] Lustre: server umount lustre-MDT0000 complete [ 586.208685] LustreError: 17381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.216472] LustreError: 17381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 588.722770] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841054 with bad export cookie 6379154825143339354 [ 588.739181] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.762044] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 589.185195] Lustre: server umount lustre-MDT0001 complete [ 596.999452] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:30:59 (1787841059) [ 600.084645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 603.982658] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:31:06 (1787841066) [ 615.466903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.665853] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.779491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 622.050837] LustreError: 19473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.019413] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 631.282892] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.335797] LustreError: 19472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 636.691204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.070318] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 658.490914] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 666.772855] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 677.567526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 687.482677] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 698.224316] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 712.172968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 712.185518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.194626] Lustre: Skipped 1 previous similar message [ 712.197383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 712.199863] Lustre: Skipped 3 previous similar messages [ 716.549109] Lustre: server umount lustre-OST0000 complete [ 722.406581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.432853] Lustre: Skipped 2 previous similar messages [ 726.977628] Lustre: server umount lustre-MDT0000 complete [ 727.527580] LustreError: 19473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.549303] LustreError: 19473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 730.939422] LustreError: 19452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841196 with bad export cookie 6379154825143340236 [ 730.949445] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.614553] Lustre: server umount lustre-MDT0001 complete [ 744.593158] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 747.470674] Key type lgssc unregistered [ 747.729399] LNet: 23011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.741406] LNetError: 23011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.765279] LNet: Removed LNI 192.168.201.116@tcp [ 748.718153] Key type .llcrypt unregistered [ 748.721184] Key type ._llcrypt unregistered [ 772.358824] Key type ._llcrypt registered [ 772.372753] Key type .llcrypt registered [ 772.561589] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 774.421538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 774.464662] alg: No test for adler32 (adler32-zlib) [ 775.984901] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 776.359271] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 778.207166] Key type lgssc registered [ 779.656723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 790.230707] loop: module loaded [ 790.316221] loop0: detected capacity change from 0 to 204800000 [ 790.580159] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 790.598347] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 791.013400] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 797.432889] loop0: detected capacity change from 0 to 204800000 [ 797.710424] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 797.721806] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 797.749773] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 803.094497] loop0: detected capacity change from 0 to 204800000 [ 809.506880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 809.527900] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.953125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 811.017642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 811.108504] Lustre: lustre-MDT0000: new disk, initializing [ 811.211071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.230720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 816.396917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 825.653989] loop1: detected capacity change from 0 to 204800000 [ 831.441509] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.879665] Lustre: lustre-OST07c6: new disk, initializing [ 831.893146] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 831.898918] Lustre: Skipped 1 previous similar message [ 831.912167] Lustre: 26204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 832.024073] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 838.779681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 838.790684] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 838.844726] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 839.508658] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 845.747435] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 845.755693] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 845.771461] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 845.778287] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 845.787478] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 845.793819] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 863.047663] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 863.058233] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 863.066645] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 863.080682] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 863.094643] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 863.104637] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 868.315450] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 868.333235] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 868.351812] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 868.363877] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.386542] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 868.397366] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.403434] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 868.410458] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.426718] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 868.433378] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 868.442923] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 868.456786] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.691569] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 871.700398] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 871.708667] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 871.714096] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.729677] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 871.736408] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.745715] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 871.755761] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.759672] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 871.764496] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 871.771496] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 871.785146] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 876.452111] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 876.460940] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 876.469905] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 876.474994] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 876.481630] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 876.489031] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 876.495120] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 876.501184] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 876.507715] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 876.513155] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 876.519707] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 876.525396] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 887.883390] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 887.891482] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 887.903060] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 887.914621] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 887.926170] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 887.931121] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 887.940287] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 887.948411] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 887.962160] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 887.981906] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 887.984785] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 887.999181] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 935.913505] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.924874] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 935.934916] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 941.027432] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 941.287445] Lustre: server umount lustre-OST07c6 complete [ 950.448175] Lustre: server umount lustre-MDT0000 complete [ 970.733896] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 974.607358] Key type lgssc unregistered [ 975.233635] LNet: 28764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.288137] LNetError: 28764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.318381] LNet: Removed LNI 192.168.201.116@tcp [ 977.032119] Key type .llcrypt unregistered [ 977.038416] Key type ._llcrypt unregistered [ 1004.323851] Key type ._llcrypt registered [ 1004.327704] Key type .llcrypt registered [ 1004.621519] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1006.513540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1006.886167] alg: No test for adler32 (adler32-zlib) [ 1008.724494] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1009.292973] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1011.111176] Key type lgssc registered [ 1013.259347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1023.233571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1023.269774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.173450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.414161] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1035.744970] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.764518] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1036.769665] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.721796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.164168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.391267] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1052.328954] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1058.661504] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1068.166130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.985128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.068303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1079.906943] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1090.371229] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1106.914283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1106.927411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.972351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.611235] Lustre: server umount lustre-OST0000 complete [ 1116.129084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.132791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.169409] Lustre: Skipped 1 previous similar message [ 1118.804883] Lustre: server umount lustre-MDT0000 complete [ 1121.253771] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.278627] LustreError: 30036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1123.519855] LustreError: 30771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841589 with bad export cookie 10580522545381346215 [ 1123.531736] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.532687] LustreError: 30771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1133.273347] Lustre: server umount lustre-MDT0001 complete [ 1144.999361] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1147.900558] Key type lgssc unregistered [ 1148.272543] LNet: 33772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.279140] LNetError: 33772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.299018] LNet: Removed LNI 192.168.201.116@tcp [ 1149.286413] Key type .llcrypt unregistered [ 1149.289302] Key type ._llcrypt unregistered [ 1166.333854] Key type ._llcrypt registered [ 1166.342408] Key type .llcrypt registered [ 1166.479335] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1167.652322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.690183] alg: No test for adler32 (adler32-zlib) [ 1168.831280] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1169.188645] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1170.999811] Key type lgssc registered [ 1172.480320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1182.464042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1182.492980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.231705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.131286] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1194.473716] LustreError: 35037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.495958] LustreError: 35037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1199.588380] LustreError: 35038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.406315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.749732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.180821] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1216.240619] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1224.598726] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1232.139740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1232.812366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.147046] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1249.176307] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1254.339906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1269.749742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.750705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1269.766359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.795847] Lustre: Skipped 1 previous similar message [ 1274.133676] Lustre: server umount lustre-OST0000 complete [ 1283.552989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1283.563692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.594236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.605704] Lustre: Skipped 1 previous similar message [ 1285.089700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1285.091288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.114462] Lustre: server umount lustre-MDT0000 complete [ 1289.249417] LustreError: 35038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.285255] LustreError: 35038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1291.160319] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841757 with bad export cookie 16945657834533894328 [ 1291.166210] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.170202] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1291.536596] Lustre: server umount lustre-MDT0001 complete [ 1302.401462] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1305.801228] Key type lgssc unregistered [ 1306.189704] LNet: 38596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.193900] LNetError: 38596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.240278] LNet: Removed LNI 192.168.201.116@tcp [ 1308.525776] Key type .llcrypt unregistered [ 1308.527598] Key type ._llcrypt unregistered [ 1334.717896] Key type ._llcrypt registered [ 1334.723168] Key type .llcrypt registered [ 1334.842802] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1335.794511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.809271] alg: No test for adler32 (adler32-zlib) [ 1336.840572] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1336.955529] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1338.584100] Key type lgssc registered [ 1339.864244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.274442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.325934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.830891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.430352] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1360.382898] LustreError: 40010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.426334] LustreError: 40010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1365.474996] LustreError: 40011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.414655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.037834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.330439] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1388.229218] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1395.884890] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1404.560098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1404.987507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.275977] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1420.355894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1420.854261] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1428.916888] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1430.499386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.511656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.533157] Lustre: Skipped 2 previous similar messages [ 1430.537736] Lustre: Skipped 1 previous similar message [ 1433.971345] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1433.981712] Lustre: Skipped 1 previous similar message [ 1434.842510] Lustre: server umount lustre-MDT0000 complete [ 1435.618671] LustreError: 42506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.646870] LustreError: 42506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1439.085253] LustreError: 42284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.437351] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841905 with bad export cookie 16950430194901933493 [ 1439.440653] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.464877] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1440.741547] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.764790] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1444.206595] LustreError: 40007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.232898] LustreError: 40007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1444.771465] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1444.776695] Lustre: Skipped 1 previous similar message [ 1446.078965] Lustre: server umount lustre-MDT0001 complete [ 1478.114037] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1478.288135] Lustre: server umount lustre-OST0000 complete [ 1497.405650] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1500.437909] Key type lgssc unregistered [ 1500.832754] LNet: 43874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.851859] LNetError: 43874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.903909] LNet: Removed LNI 192.168.201.116@tcp [ 1501.586638] Key type .llcrypt unregistered [ 1501.589282] Key type ._llcrypt unregistered [ 1527.511599] Key type ._llcrypt registered [ 1527.514963] Key type .llcrypt registered [ 1527.629635] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1528.542338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1528.670029] alg: No test for adler32 (adler32-zlib) [ 1529.794825] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1530.069240] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1531.815168] Key type lgssc registered [ 1533.246747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.566212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1541.600946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.040235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.352781] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1553.390847] LustreError: 45288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.434447] LustreError: 45288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1557.563264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.100815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1562.707800] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1570.647689] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1576.371448] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1583.163715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1583.717185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.841466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1590.532375] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1599.107233] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1609.701229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.713076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.736414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1612.196679] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1612.775866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1612.782412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.702339] Lustre: server umount lustre-OST0000 complete [ 1634.804127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1634.817134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.839636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1634.851444] Lustre: Skipped 2 previous similar messages [ 1637.344428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.347646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.369833] Lustre: Skipped 1 previous similar message [ 1642.471303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.485383] Lustre: Skipped 1 previous similar message [ 1649.119978] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1649.377286] Lustre: server umount lustre-MDT0000 complete [ 1652.712471] LustreError: 45287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.734214] LustreError: 45287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1653.702540] LustreError: 45268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842119 with bad export cookie 1122175030609674540 [ 1653.718382] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1653.720501] LustreError: 45268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1668.063329] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1668.414322] Lustre: server umount lustre-MDT0001 complete [ 1681.719753] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1684.752903] Key type lgssc unregistered [ 1685.082972] LNet: 48998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.094600] LNetError: 48998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.115306] LNet: Removed LNI 192.168.201.116@tcp [ 1686.227529] Key type .llcrypt unregistered [ 1686.231763] Key type ._llcrypt unregistered [ 1714.521383] Key type ._llcrypt registered [ 1714.523607] Key type .llcrypt registered [ 1714.659606] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1716.248316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.277288] alg: No test for adler32 (adler32-zlib) [ 1717.396378] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1717.806279] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1719.599293] Key type lgssc registered [ 1720.792697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1729.200964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1729.217420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.731558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.248466] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1740.260814] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.285377] LustreError: 50414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1744.238520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.749276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1750.517326] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1758.256614] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1765.307765] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1772.725411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.197566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.217040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1779.821628] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1787.382809] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1796.205838] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1799.898702] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1812.089538] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2710h51m15s [ 1813.509874] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1815.522150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.526477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.537471] Lustre: Skipped 1 previous similar message [ 1816.612874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.621850] Lustre: Skipped 3 previous similar messages [ 1820.027501] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1821.846186] Lustre: server umount lustre-MDT0000 complete [ 1822.697175] LustreError: 50408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.713863] LustreError: 50408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1826.187635] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1827.179071] LustreError: 50408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.356196] LustreError: 53107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.378061] LustreError: 53107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1833.810506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.901533] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1834.193579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.265094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1835.253037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1838.919051] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1842.142922] Lustre: DEBUG MARKER: Wait for df (36674) ... [ 1843.505559] Lustre: DEBUG MARKER: done [ 1849.831897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.838989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.848384] Lustre: Skipped 1 previous similar message [ 1849.852491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.859574] Lustre: Skipped 1 previous similar message [ 1850.850946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.066560] Lustre: server umount lustre-OST0000 complete [ 1861.089238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1861.089585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.104442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.104451] Lustre: Skipped 2 previous similar messages [ 1861.148198] Lustre: Skipped 1 previous similar message [ 1871.327121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1871.338789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.343719] Lustre: Skipped 5 previous similar messages [ 1871.631739] Lustre: server umount lustre-MDT0000 complete [ 1875.900746] LustreError: 50392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842341 with bad export cookie 515469102499117856 [ 1875.903528] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.909599] LustreError: 50392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1876.279398] Lustre: server umount lustre-MDT0001 complete [ 1888.628930] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1891.836959] Key type lgssc unregistered [ 1892.140095] LNet: 55335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.150431] LNetError: 55335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.179484] LNet: Removed LNI 192.168.201.116@tcp [ 1893.062146] Key type .llcrypt unregistered [ 1893.064684] Key type ._llcrypt unregistered [ 1940.180068] Key type ._llcrypt registered [ 1940.184962] Key type .llcrypt registered [ 1940.293626] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1942.028155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.134771] alg: No test for adler32 (adler32-zlib) [ 1943.332705] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1943.678961] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1945.383197] Key type lgssc registered [ 1946.784614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1956.554610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1956.617455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.060982] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1958.085818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1958.258536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.762733] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1973.613433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.714517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1973.751483] Lustre: 57541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1974.124336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.373455] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1987.841520] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1994.197877] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2001.630373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2001.812935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2001.965604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.289894] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2009.321143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2017.442373] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.576437] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2710h54m48s [ 2026.325315] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2033.039979] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2035.574352] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2035.794625] LustreError: 58294:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2056.164581] Lustre: lustre-MDT0000: Client faa364d7-185e-4865-99dc-4da3621fb7f7 (at 192.168.201.16@tcp) reconnecting [ 2075.815234] LustreError: 58294:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2078.305652] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2080.880745] Lustre: DEBUG MARKER: done [ 2091.489210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.492972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.504897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.508136] Lustre: Skipped 1 previous similar message [ 2094.867413] Lustre: server umount lustre-OST0000 complete [ 2101.731075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.752858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.757635] Lustre: Skipped 1 previous similar message [ 2102.819688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.825967] Lustre: Skipped 1 previous similar message [ 2104.585370] Lustre: server umount lustre-MDT0000 complete [ 2108.211696] LustreError: 57533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842574 with bad export cookie 14450595573782067601 [ 2108.213861] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2108.229370] LustreError: 57533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2108.632703] Lustre: server umount lustre-MDT0001 complete [ 2121.139683] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2124.173811] Key type lgssc unregistered [ 2124.468899] LNet: 61869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.479799] LNetError: 61869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.509683] LNet: Removed LNI 192.168.201.116@tcp [ 2125.627948] Key type .llcrypt unregistered [ 2125.629845] Key type ._llcrypt unregistered [ 2168.133315] Key type ._llcrypt registered [ 2168.135525] Key type .llcrypt registered [ 2168.225847] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2169.209286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.284406] alg: No test for adler32 (adler32-zlib) [ 2170.325718] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 2170.525378] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2172.183936] Key type lgssc registered [ 2173.566941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2176.761913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.778485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.149646] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2178.176712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2178.384344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.522615] Lustre: Failing over lustre-MDT0000 [ 2179.804663] Lustre: server umount lustre-MDT0000 complete [ 2186.371858] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:57:30 (1787842650) [ 2192.978109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.361452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.138813] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2205.983489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.081433] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2206.115824] Lustre: 64661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2206.432436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.297475] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2219.361281] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2224.904190] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2232.167715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2232.385652] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2232.536849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.260382] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2242.766548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2246.068843] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2252.179950] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2257.889615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.894948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.900480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.406946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.413216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.438017] Lustre: Skipped 1 previous similar message [ 2263.520537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.527395] Lustre: Skipped 2 previous similar messages [ 2264.070408] Lustre: server umount lustre-MDT0000 complete [ 2268.252537] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842734 with bad export cookie 1652539055628530586 [ 2268.270063] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.277551] LustreError: 64651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2268.609553] Lustre: server umount lustre-MDT0001 complete [ 2271.869077] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2276.239913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2284.003880] Lustre: 63582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842734/real 1787842734] req@00000000258b116a x1874688879924224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2284.035875] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.283298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.183642] Lustre: 63582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842739/real 1787842739] req@00000000bd2125f2 x1874688879924352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2293.284641] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x16eeff83fc32bc84 [ 2293.297200] Lustre: MGC192.168.201.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2293.578567] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.600326] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2293.664106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.759869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2294.634414] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842744/real 1787842744] req@000000000718f617 x1874688879924608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2294.692964] LustreError: 68431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.722428] LustreError: 68431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2298.326963] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2299.881988] LustreError: 68427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.914842] LustreError: 68427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2300.895155] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842750/real 1787842750] req@0000000034c8425c x1874688879924736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2305.020303] LustreError: 68524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.042174] LustreError: 68524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2307.391858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.865181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.388940] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2313.002215] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2319.275280] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.028912] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2333.667863] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.670411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.677835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.677845] Lustre: Skipped 2 previous similar messages [ 2333.684199] Lustre: Skipped 1 previous similar message [ 2336.246692] Lustre: server umount lustre-MDT0000 complete [ 2338.784393] LustreError: 68524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.806474] LustreError: 68524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2339.648921] LustreError: 67571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842805 with bad export cookie 1652539055628532868 [ 2339.658968] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.665775] LustreError: 67571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2339.920566] Lustre: server umount lustre-MDT0001 complete [ 2346.490475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2357.123609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.199543] Lustre: 63582:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842809/real 1787842809] req@00000000fe77e5aa x1874688879950848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2359.212337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.220088] Lustre: Skipped 1 previous similar message [ 2364.387037] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x16eeff83fc32c1e7 [ 2364.397294] Lustre: MGC192.168.201.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2364.405659] Lustre: Skipped 1 previous similar message [ 2364.825704] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.931257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.989669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2369.110234] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2370.982449] Lustre: 63581:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842820/real 1787842820] req@0000000087bc5196 x1874688879951488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2371.009054] Lustre: 63581:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2377.477923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.672082] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2383.344215] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2388.522631] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2390.036519] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2395.257710] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.297720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2403.313920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.331603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.334290] Lustre: Skipped 2 previous similar messages [ 2408.356725] Lustre: server umount lustre-MDT0000 complete [ 2408.929447] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.946380] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2411.594942] LustreError: 64652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842877 with bad export cookie 1652539055628534247 [ 2411.605175] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.609792] LustreError: 64652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2411.895042] Lustre: server umount lustre-MDT0001 complete [ 2417.616075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2420.667491] Lustre: DEBUG MARKER: files compared the same [ 2432.923548] Lustre: server umount lustre-OST0000 complete [ 2449.686903] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2451.836286] Key type lgssc unregistered [ 2452.163502] LNet: 74400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.174790] LNetError: 74400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2453.224471] LNet: Removed LNI 192.168.201.116@tcp [ 2454.300453] Key type .llcrypt unregistered [ 2454.306542] Key type ._llcrypt unregistered [ 2475.966623] Key type ._llcrypt registered [ 2475.968830] Key type .llcrypt registered [ 2476.066870] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2477.281904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.361922] alg: No test for adler32 (adler32-zlib) [ 2478.493699] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 2478.757580] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2480.455233] Key type lgssc registered [ 2481.705738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2491.150468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.178349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.682537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.239829] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2502.113222] LustreError: 75794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.129858] LustreError: 75794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2507.236725] LustreError: 75793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.292210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.515422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.692438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2521.703240] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2528.873162] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2537.545741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.088575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.118887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2544.800770] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2553.805581] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2564.068101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.070439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2564.075349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.090946] Lustre: Skipped 1 previous similar message [ 2569.185013] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.191101] Lustre: Skipped 2 previous similar messages [ 2569.851445] Lustre: server umount lustre-OST0000 complete [ 2574.304289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.311287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.311517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.316629] Lustre: Skipped 1 previous similar message [ 2579.424418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.433547] Lustre: Skipped 2 previous similar messages [ 2580.603453] Lustre: server umount lustre-MDT0000 complete [ 2584.552539] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.561060] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2584.809490] LustreError: 75774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843050 with bad export cookie 14750635851742011479 [ 2584.812292] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.822547] LustreError: 75774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2585.213440] Lustre: server umount lustre-MDT0001 complete [ 2596.029825] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2599.040362] Key type lgssc unregistered [ 2599.281435] LNet: 79327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2599.292477] LNetError: 79327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2599.317633] LNet: Removed LNI 192.168.201.116@tcp [ 2600.153262] Key type .llcrypt unregistered [ 2600.155508] Key type ._llcrypt unregistered [ 2628.374250] Key type ._llcrypt registered [ 2628.382303] Key type .llcrypt registered [ 2628.506814] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2629.768727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2629.891379] alg: No test for adler32 (adler32-zlib) [ 2631.167435] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 2631.406410] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2633.071311] Key type lgssc registered [ 2634.499731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.631625] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.664526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.271612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2663.392416] LustreError: 80714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 2690.080115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.154036] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2704.672938] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2712.377818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.538464] Lustre: *** cfs_fail_loc=706, val=0*** [ 2712.948588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.034362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2716.645647] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2718.193934] LustreError: 80739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.214544] LustreError: 80739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2723.297718] LustreError: 80740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.322947] LustreError: 80739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.343887] LustreError: 80739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2724.621465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.048487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.712894] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2737.843276] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2743.466566] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2745.842731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.855306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2745.861184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.952101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.959281] Lustre: Skipped 2 previous similar messages [ 2756.071088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.097265] Lustre: Skipped 1 previous similar message [ 2760.159253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2760.355557] Lustre: server umount lustre-OST0000 complete [ 2766.305893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.308482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.310693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.320260] Lustre: Skipped 2 previous similar messages [ 2771.147643] Lustre: server umount lustre-MDT0000 complete [ 2771.435341] LustreError: 81805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.502968] LustreError: 81805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2776.327352] LustreError: 80720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843242 with bad export cookie 4032660431936826251 [ 2776.333783] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.337887] LustreError: 80720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2776.698355] Lustre: server umount lustre-MDT0001 complete [ 2789.249850] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2792.262394] Key type lgssc unregistered [ 2792.569572] LNet: 84333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.575130] LNetError: 84333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.601935] LNet: Removed LNI 192.168.201.116@tcp [ 2793.638894] Key type .llcrypt unregistered [ 2793.641742] Key type ._llcrypt unregistered [ 2817.757810] Key type ._llcrypt registered [ 2817.763694] Key type .llcrypt registered [ 2817.863715] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2819.046370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2819.065762] alg: No test for adler32 (adler32-zlib) [ 2820.178295] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 2820.448626] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2822.135176] Key type lgssc registered [ 2823.160435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2829.825477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2829.838780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.557642] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2842.018332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.687310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.627959] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2847.713576] LustreError: 86433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.729696] LustreError: 86433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2852.835750] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.853684] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2856.817058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.264968] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.298074] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2857.307686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.570822] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.589542] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2864.578932] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2867.702642] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.732810] LustreError: 86436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2873.305100] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.326355] Lustre: MGS: Client 9f0c944d-41f9-4a5d-8b1b-cf69a4c7deb4 (at 0@lo) reconnecting [ 2873.364454] LustreError: 86447:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2873.685785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.802296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2880.467933] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2894.315599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.328962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2894.345162] Lustre: Skipped 1 previous similar message [ 2894.357269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2899.383323] Lustre: server umount lustre-OST0000 complete [ 2904.549877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.550316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2904.574105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.585245] Lustre: Skipped 2 previous similar messages [ 2909.536550] Lustre: server umount lustre-MDT0000 complete [ 2909.671113] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.689235] LustreError: 86437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2913.115102] Lustre: server umount lustre-MDT0001 complete [ 2917.421674] Lustre: server umount MGS complete [ 2928.515884] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:09:52 (1787843392) [ 2958.835277] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 2967.993399] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3007.761837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.839585] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3022.921462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.228737] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3023.273703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3023.376425] Lustre: lustre-MDT0001: new disk, initializing [ 3023.472117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.518376] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3023.529259] Lustre: 93601:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3023.544983] LustreError: 93601:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3028.307845] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3039.305229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.570887] Lustre: lustre-OST0000: new disk, initializing [ 3039.578673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3039.589762] Lustre: 94367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3039.681613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.567205] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3050.099080] Lustre: 94824:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000070d1454c x1874689561501952/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787843532 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3056.742960] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3064.635366] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.649939] Lustre: MGS: Client 80e646be-7ad7-4185-b9e0-f0d7fcb95ba1 (at 0@lo) reconnecting [ 3064.725913] Lustre: 92868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3064.786986] LustreError: 93587:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3064.840101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3064.969464] Lustre: lustre-MDT0000: new disk, initializing [ 3065.148812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.196873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3065.230206] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3066.276854] Lustre: 95437:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000002c113a68 x1874689561508992/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787843548 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3070.056273] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3071.403156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3071.412931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3071.560275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3096.575987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.583156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3096.596490] Lustre: Skipped 2 previous similar messages [ 3100.614800] Lustre: server umount lustre-OST0000 complete [ 3106.784248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3106.788504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.798713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.798720] Lustre: Skipped 1 previous similar message [ 3106.827056] Lustre: Skipped 1 previous similar message [ 3110.279675] Lustre: server umount lustre-MDT0000 complete [ 3111.905071] LustreError: 93572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.929021] LustreError: 93572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3120.533117] Lustre: server umount lustre-MDT0001 complete [ 3124.269488] Lustre: server umount MGS complete [ 3135.187908] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:13:19 (1787843599) [ 3154.224431] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3157.237377] Key type lgssc unregistered [ 3157.454381] LNet: 97933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3157.462953] LNetError: 97933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.482274] LNet: Removed LNI 192.168.201.116@tcp [ 3158.356697] Key type .llcrypt unregistered [ 3158.358857] Key type ._llcrypt unregistered [ 3178.397374] Key type ._llcrypt registered [ 3178.399756] Key type .llcrypt registered [ 3178.560576] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3180.105506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3180.189793] alg: No test for adler32 (adler32-zlib) [ 3181.325390] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3181.743633] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3183.431220] Key type lgssc registered [ 3185.008334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.690903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.707748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.159393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.872679] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3205.604219] LustreError: 99153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.619170] LustreError: 99153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3208.317484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.599909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3213.140217] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3220.866969] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3227.287617] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3235.768137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.147406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.278748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3243.228532] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3251.639403] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3260.897530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.907896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.927788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.980308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.989586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3263.992865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.903683] Lustre: server umount lustre-OST0000 complete [ 3274.213315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.228744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.235896] Lustre: Skipped 2 previous similar messages [ 3276.547520] Lustre: server umount lustre-MDT0000 complete [ 3279.329700] LustreError: 99148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.353638] LustreError: 99148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3279.652991] LustreError: 99134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843745 with bad export cookie 5559096519467861857 [ 3279.662032] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.930213] Lustre: server umount lustre-MDT0001 complete [ 3291.567452] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3294.226671] Key type lgssc unregistered [ 3294.424806] LNet: 102691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3294.434363] LNetError: 102691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3294.447258] LNet: Removed LNI 192.168.201.116@tcp [ 3295.370798] Key type .llcrypt unregistered [ 3295.373234] Key type ._llcrypt unregistered [ 3311.118582] Key type ._llcrypt registered [ 3311.120589] Key type .llcrypt registered [ 3311.220867] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3312.420381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3312.524637] alg: No test for adler32 (adler32-zlib) [ 3313.538840] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3313.778712] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3315.447187] Key type lgssc registered [ 3316.469741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3322.525502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3322.527856] LustreError: 103770:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3322.530820] Lustre: Skipped 1 previous similar message [ 3322.530842] LustreError: 103771:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3322.545877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.569797] LustreError: 103770:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3322.581751] LustreError: 103770:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3322.601615] LustreError: 103770:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3324.050769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.821584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.019581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3335.028784] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3339.868741] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing lsmod [ 3342.518817] LustreError: 104973:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3342.534551] LustreError: 104977:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3342.541674] LustreError: 104973:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 3342.552162] LustreError: 104973:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3342.563443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.563658] LustreError: 104973:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3342.910335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.401129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3346.414183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.433161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.939353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.940594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.955332] Lustre: Skipped 1 previous similar message [ 3347.977500] Lustre: Skipped 1 previous similar message [ 3352.460389] Lustre: server umount lustre-MDT0000 complete [ 3353.060159] LustreError: 103803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.087371] LustreError: 103803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3355.121636] LustreError: 103805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.146839] LustreError: 103805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3355.876241] LustreError: 103784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843821 with bad export cookie 143376237108226548 [ 3355.883470] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.215390] LustreError: 104240:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002a747106 x1874690078429056/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3357.382438] Lustre: server umount lustre-MDT0001 complete [ 3370.390762] Lustre: server umount lustre-OST0000 complete [ 3379.601767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.964397] LustreError: 106040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.045775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.328169] LustreError: 106041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.361764] LustreError: 106041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3385.449523] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3389.412942] LustreError: 106040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.872280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.108112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3399.415957] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3407.429392] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3413.490283] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3419.269878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.715774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.823949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3425.823634] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3433.935151] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3445.216652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.221563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.232839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.237270] Lustre: Skipped 1 previous similar message [ 3445.245087] Lustre: Skipped 1 previous similar message [ 3447.752136] Lustre: server umount lustre-OST0000 complete [ 3455.457133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.480448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.487231] Lustre: Skipped 2 previous similar messages [ 3458.170290] Lustre: server umount lustre-MDT0000 complete [ 3460.576860] LustreError: 106036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.597376] LustreError: 106036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3461.834149] LustreError: 106020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843927 with bad export cookie 143376237108227234 [ 3461.838491] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.844064] LustreError: 106020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3462.315035] Lustre: server umount lustre-MDT0001 complete [ 3474.408424] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3478.226610] Key type lgssc unregistered [ 3478.583829] LNet: 109577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3478.593741] LNetError: 109577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3478.615844] LNet: Removed LNI 192.168.201.116@tcp [ 3480.070884] Key type .llcrypt unregistered [ 3480.074625] Key type ._llcrypt unregistered [ 3509.955193] Key type ._llcrypt registered [ 3509.959801] Key type .llcrypt registered [ 3510.179829] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3512.037619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3512.060703] alg: No test for adler32 (adler32-zlib) [ 3513.175847] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3513.459584] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3515.200326] Key type lgssc registered [ 3516.161614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.793707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3524.841333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.565062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.561407] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3536.865827] LustreError: 110993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.899151] LustreError: 110993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3537.888177] LustreError: 110994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.195855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.610660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3546.653602] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3554.745850] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3561.849812] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3570.285464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.646685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.751102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3578.495980] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3588.651366] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.545078] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3598.459875] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3603.424587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.430503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.453471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.513089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3606.517934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.519778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.539400] Lustre: Skipped 1 previous similar message [ 3609.309744] Lustre: server umount lustre-OST0000 complete [ 3613.152983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3613.155865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.186724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.739184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.746738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.975742] Lustre: server umount lustre-MDT0000 complete [ 3621.857216] LustreError: 110993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.866202] LustreError: 110993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3623.124074] LustreError: 110974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844089 with bad export cookie 14328640022955376989 [ 3623.135831] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.140835] LustreError: 110974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3623.487235] Lustre: server umount lustre-MDT0001 complete [ 3636.435460] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3639.157535] Key type lgssc unregistered [ 3639.440869] LNet: 114634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3639.449186] LNetError: 114634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3640.491760] LNet: Removed LNI 192.168.201.116@tcp [ 3641.628595] Key type .llcrypt unregistered [ 3641.630759] Key type ._llcrypt unregistered [ 3658.528307] Key type ._llcrypt registered [ 3658.540990] Key type .llcrypt registered [ 3658.699526] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3659.998549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3660.059125] alg: No test for adler32 (adler32-zlib) [ 3661.269835] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3661.539629] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3663.199987] Key type lgssc registered [ 3664.444182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3672.076707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3672.091254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.454595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.695598] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3683.819785] LustreError: 115850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.846399] LustreError: 115850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3685.846402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.141313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.548459] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3698.133105] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3703.894977] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3710.625604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.094703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.757121] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3718.294583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3726.329789] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3738.094425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3738.107600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.125022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3739.107363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3739.111474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3739.131340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.227306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.237064] Lustre: Skipped 2 previous similar messages [ 3748.831847] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3748.943912] Lustre: server umount lustre-OST0000 complete [ 3754.464758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.485586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3754.491100] Lustre: Skipped 1 previous similar message [ 3759.590536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.593987] Lustre: Skipped 1 previous similar message [ 3767.264281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3767.529516] Lustre: server umount lustre-MDT0000 complete [ 3769.825911] LustreError: 115847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.847831] LustreError: 115847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3771.323650] LustreError: 115831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844237 with bad export cookie 7793692012015631269 [ 3771.337640] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.825416] Lustre: server umount lustre-MDT0001 complete [ 3785.112065] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3787.512587] Key type lgssc unregistered [ 3787.791081] LNet: 119387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3787.801336] LNetError: 119387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3787.819310] LNet: Removed LNI 192.168.201.116@tcp [ 3788.665881] Key type .llcrypt unregistered [ 3788.668895] Key type ._llcrypt unregistered [ 3813.970518] Key type ._llcrypt registered [ 3813.972633] Key type .llcrypt registered [ 3814.157653] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3815.568110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3815.818744] alg: No test for adler32 (adler32-zlib) [ 3817.064254] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3817.367844] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3819.135244] Key type lgssc registered [ 3820.284691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3829.269174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3829.306786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.219303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.248245] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3841.517619] LustreError: 120800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3841.561399] LustreError: 120800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3845.127876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.616395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3850.542059] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3858.442099] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3863.785541] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3870.721426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3870.981656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.148660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3877.067670] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3884.639385] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3895.477683] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3899.086172] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3899.094108] Lustre: Skipped 2 previous similar messages [ 3901.026784] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3901.093892] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3903.088249] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3903.100604] Lustre: Skipped 2 previous similar messages [ 3910.259353] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3910.282189] Lustre: Skipped 1 previous similar message [ 3915.325769] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3915.344968] Lustre: Skipped 2 previous similar messages [ 3920.027196] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.16@tcp 192.168.0.[2,10]@tcp [ 3925.209507] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.16/24@tcp in log lustre-MDT0000 [ 3925.219806] Lustre: Skipped 2 previous similar messages [ 3926.146425] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.16/24@tcp [ 3926.163253] Lustre: Skipped 1 previous similar message [ 3932.652865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.660199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.670287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.746363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.748546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.759724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.789935] Lustre: Skipped 1 previous similar message [ 3938.539593] Lustre: server umount lustre-OST0000 complete [ 3942.889169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3942.894986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.910505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.977817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.980600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.826700] Lustre: server umount lustre-MDT0000 complete [ 3952.099028] LustreError: 120795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.111906] LustreError: 120795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3952.700428] LustreError: 120779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844418 with bad export cookie 16778967937789189939 [ 3952.700428] LustreError: 121535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844418 with bad export cookie 16778967937789189939 [ 3952.700745] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.164614] Lustre: server umount lustre-MDT0001 complete [ 3964.901525] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3967.757595] Key type lgssc unregistered [ 3968.126092] LNet: 125599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3968.134233] LNetError: 125599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3968.159043] LNet: Removed LNI 192.168.201.116@tcp [ 3969.175163] Key type .llcrypt unregistered [ 3969.176825] Key type ._llcrypt unregistered [ 3996.505541] Key type ._llcrypt registered [ 3996.508560] Key type .llcrypt registered [ 3996.671379] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3998.316303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3998.391125] alg: No test for adler32 (adler32-zlib) [ 3999.523748] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3999.759358] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4001.479245] Key type lgssc registered [ 4002.431574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4010.603524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4010.623491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.092476] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4012.129903] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.16/24@tcp [ 4012.130526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.369876] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4022.242225] LustreError: 127159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.268561] LustreError: 127159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4025.024103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4025.428384] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4025.492489] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.16/24@tcp [ 4025.531483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4031.206995] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4039.173340] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4046.220385] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4054.520233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4055.264753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.354391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4063.149299] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4071.607822] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4083.170185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.185391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.206696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.240137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.250875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.258323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.276297] Lustre: Skipped 1 previous similar message [ 4089.080470] Lustre: server umount lustre-OST0000 complete [ 4096.481306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.483747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.488148] Lustre: Skipped 1 previous similar message [ 4096.498827] Lustre: Skipped 1 previous similar message [ 4099.738451] Lustre: server umount lustre-MDT0000 complete [ 4101.603306] LustreError: 127155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.622136] LustreError: 127155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4103.667184] LustreError: 127140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844569 with bad export cookie 16243138934731835120 [ 4103.672969] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.677179] LustreError: 127140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4104.049571] Lustre: server umount lustre-MDT0001 complete [ 4115.103962] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4117.942884] Key type lgssc unregistered [ 4118.365057] LNet: 130745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4118.377415] LNetError: 130745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.401556] LNet: Removed LNI 192.168.201.116@tcp [ 4119.354754] Key type .llcrypt unregistered [ 4119.356653] Key type ._llcrypt unregistered [ 4151.263216] Key type ._llcrypt registered [ 4151.266536] Key type .llcrypt registered [ 4151.405104] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4169.125556] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4170.704628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4170.800954] alg: No test for adler32 (adler32-zlib) [ 4171.962141] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4172.258517] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4173.927185] Key type lgssc registered [ 4175.276696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4215.378256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4215.429937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.907662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4216.960867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4217.089904] Lustre: lustre-MDT0000: new disk, initializing [ 4217.243812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.269386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4222.771743] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4234.800049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.980468] Lustre: 134144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4235.057872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4235.063373] Lustre: Skipped 1 previous similar message [ 4235.184849] Lustre: lustre-MDT0001: new disk, initializing [ 4235.318673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4235.389407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4235.410925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4239.829521] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4249.453307] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4255.723191] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4261.936855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4262.350863] Lustre: lustre-OST0000: new disk, initializing [ 4262.356554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4262.368791] Lustre: 136113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4262.477187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.068712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4268.088894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4268.230186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4268.429513] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4278.676412] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4283.974872] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4284.199855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4289.659918] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4289.921787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4294.125496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.136319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.138083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.159812] Lustre: Skipped 1 previous similar message [ 4294.181425] Lustre: Skipped 1 previous similar message [ 4298.963184] Lustre: server umount lustre-OST0000 complete [ 4304.361485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.380174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.385863] Lustre: Skipped 1 previous similar message [ 4309.474160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4309.479042] Lustre: Skipped 1 previous similar message [ 4314.596321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.607508] Lustre: Skipped 1 previous similar message [ 4318.175955] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4318.679302] Lustre: server umount lustre-MDT0000 complete [ 4322.565809] LustreError: 134136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844788 with bad export cookie 13017645539854088238 [ 4322.573617] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.576236] LustreError: 134136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4322.953585] Lustre: server umount lustre-MDT0001 complete [ 4333.801620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.421984] LustreError: 138333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.439953] LustreError: 138333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4334.515939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.682424] LustreError: 138334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.699204] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4344.803678] LustreError: 138333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.951685] LustreError: 138334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.720597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.286976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4356.934858] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4367.220939] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4373.806319] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4383.048332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.534518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.140410] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4399.622890] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4406.026549] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4406.432619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4413.885287] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4414.128678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4425.515239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4425.685519] Lustre: lustre-OST0001: new disk, initializing [ 4425.688594] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4425.695357] Lustre: 141707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4425.813456] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.527918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4432.541187] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4432.667625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4433.306970] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4441.831690] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4442.052883] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4446.818858] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4447.013833] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4454.185986] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91bfc8d36800.ost_server_uuid 50 [ 4456.162772] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91bfc8d36800.ost_server_uuid in IDLE state after 0 sec [ 4468.715818] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.726360] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.736613] Lustre: Skipped 1 previous similar message [ 4468.745688] LustreError: Skipped 2 previous similar messages [ 4468.746458] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4468.768204] Lustre: Skipped 1 previous similar message [ 4472.648564] Lustre: server umount lustre-OST0001 complete [ 4473.837699] LustreError: 142772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.850601] LustreError: 142772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4478.944403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4478.951191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.962552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4478.978380] Lustre: Skipped 1 previous similar message [ 4478.999827] Lustre: Skipped 1 previous similar message [ 4483.665451] Lustre: server umount lustre-OST0000 complete [ 4489.190753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.306316] LustreError: 138333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.332995] LustreError: 138333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4494.588831] Lustre: server umount lustre-MDT0000 complete [ 4499.201947] LustreError: 139067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844965 with bad export cookie 13017645539854089127 [ 4499.224670] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4499.228899] LustreError: 139067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4500.036506] Lustre: server umount lustre-MDT0001 complete [ 4523.932931] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4527.621295] Key type lgssc unregistered [ 4528.032952] LNet: 144716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4528.044317] LNetError: 144716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4528.068981] LNet: Removed LNI 192.168.201.116@tcp [ 4529.542341] Key type .llcrypt unregistered [ 4529.545611] Key type ._llcrypt unregistered [ 4576.556690] Key type ._llcrypt registered [ 4576.558636] Key type .llcrypt registered [ 4576.674805] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4599.196886] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4600.214327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4600.292445] alg: No test for adler32 (adler32-zlib) [ 4601.327221] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4601.508147] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4603.279895] Key type lgssc registered [ 4605.589716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4648.939165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4648.993867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.489401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4650.602919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4650.750538] Lustre: lustre-MDT0000: new disk, initializing [ 4650.883648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.913934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4656.355366] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4668.848932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.009272] Lustre: 148872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4669.069899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4669.074866] Lustre: Skipped 1 previous similar message [ 4669.163257] Lustre: lustre-MDT0001: new disk, initializing [ 4669.320508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4669.365819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4669.369954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4674.461734] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4685.057349] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4691.005856] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4698.413254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4698.857529] Lustre: lustre-OST0000: new disk, initializing [ 4698.878531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4698.890542] Lustre: 150841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4699.054499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.833808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4705.848316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4705.997038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4706.350580] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4715.442491] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4719.223098] Lustre: Failing over lustre-OST0000 [ 4719.552370] Lustre: server umount lustre-OST0000 complete [ 4736.992464] Lustre: 147091:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845187/real 1787845187] req@000000004e367d62 x1874691429009280/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787845203 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4736.993034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.018934] Lustre: 147091:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4740.255576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4740.643880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.663854] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4743.648591] Lustre: 147094:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845192/real 1787845192] req@000000004b4fda0a x1874691429010944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4743.686507] Lustre: 147094:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4747.594917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4747.745195] Lustre: 147091:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845197/real 1787845197] req@00000000d4b912a0 x1874691429012224/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4747.772196] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4747.793964] Lustre: 147091:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4748.714852] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4748.717867] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4749.751208] Lustre: 147094:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845197/real 1787845197] req@0000000073d3ccd1 x1874691429012096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787845213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4751.337811] Lustre: Failing over lustre-MDT0000 [ 4751.885395] Lustre: server umount lustre-MDT0000 complete [ 4752.882366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.898088] Lustre: Skipped 2 previous similar messages [ 4757.987469] LustreError: 148878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.006377] LustreError: 148878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4758.903960] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.106809] LustreError: 148879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.122503] LustreError: 148879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4768.095934] Lustre: 147093:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845218/real 1787845218] req@000000003e933f39 x1874691429022848/t0(0) o400->MGC192.168.201.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787845234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4768.127265] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4768.145714] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.173671] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4772.390203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.346730] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.356817] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4778.838088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.890166] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4780.234148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4784.116745] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4784.123462] Lustre: Skipped 1 previous similar message [ 4784.164084] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4784.220955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4784.450388] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4794.340504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.345110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4794.352821] Lustre: Skipped 1 previous similar message [ 4794.366637] Lustre: Skipped 1 previous similar message [ 4799.268633] Lustre: server umount lustre-OST0000 complete [ 4804.603370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.609242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4804.620719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4804.645731] Lustre: Skipped 1 previous similar message [ 4809.696873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4809.707793] Lustre: Skipped 1 previous similar message [ 4810.629502] Lustre: server umount lustre-MDT0000 complete [ 4814.818621] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.838860] LustreError: 148883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4815.321951] LustreError: 148865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845281 with bad export cookie 14844666503412449308 [ 4815.340621] LustreError: 148865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4815.342321] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.313994] Lustre: server umount lustre-MDT0001 complete [ 4832.347507] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4835.947383] Key type lgssc unregistered [ 4836.344271] LNet: 154501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4836.353507] LNetError: 154501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4836.382416] LNet: Removed LNI 192.168.201.116@tcp [ 4837.585231] Key type .llcrypt unregistered [ 4837.586905] Key type ._llcrypt unregistered [ 4877.949607] Key type ._llcrypt registered [ 4877.956956] Key type .llcrypt registered [ 4878.102838] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4879.536082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4879.712883] alg: No test for adler32 (adler32-zlib) [ 4880.917515] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4881.267864] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4883.015181] Key type lgssc registered [ 4884.201296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4891.401637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4891.423148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.096371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.990122] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4903.397091] LustreError: 156018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.415117] LustreError: 156018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4908.377648] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4908.514662] LustreError: 156019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.885206] LustreError: 156047:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4912.896676] LustreError: 156047:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4912.908840] LustreError: 156047:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4913.392953] Lustre: server umount lustre-MDT0000 complete [ 4929.860630] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4940.825189] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4986.772477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.125500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4987.163743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4987.291555] Lustre: lustre-MDT0000: new disk, initializing [ 4987.428183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4987.511302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4994.913835] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5003.903921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.018041] Lustre: 159351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5004.088609] LustreError: 159363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.110206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5004.110215] Lustre: Skipped 1 previous similar message [ 5004.197984] Lustre: lustre-MDT0001: new disk, initializing [ 5004.280969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.332823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5004.345025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5010.982943] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5020.836078] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5028.107677] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5035.128417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5035.487852] Lustre: lustre-OST0000: new disk, initializing [ 5035.495484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5035.500922] Lustre: 161323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5035.590233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5042.855763] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5044.817149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5044.825578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5044.924945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5053.357402] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5058.019948] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5060.169906] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:45:24 (1787845524) [ 5270.499498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5270.502496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.503616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.533365] Lustre: Skipped 1 previous similar message [ 5274.498467] Lustre: server umount lustre-MDT0000 complete [ 5275.618725] LustreError: 159359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.650058] LustreError: 159359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5278.704634] LustreError: 159345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845744 with bad export cookie 8612102818445259081 [ 5278.705797] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5278.711973] LustreError: 159345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5280.744564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.747933] LustreError: 159360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.766573] Lustre: Skipped 1 previous similar message [ 5280.767842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5280.767848] Lustre: Skipped 2 previous similar messages [ 5283.810933] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5288.931162] LustreError: 160897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.931845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5288.952538] LustreError: 160897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5293.024894] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5293.561919] Lustre: server umount lustre-MDT0001 complete [ 5304.105796] Lustre: server umount lustre-OST0000 complete [ 5316.481057] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5327.937832] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5367.076991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.399204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5367.443691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5367.567644] Lustre: lustre-MDT0000: new disk, initializing [ 5367.689082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.720435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5372.102153] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5384.074477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.298800] Lustre: 165980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5384.410137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5384.423739] Lustre: Skipped 1 previous similar message [ 5384.578249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5384.617669] Lustre: lustre-MDT0001: new disk, initializing [ 5384.701555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5384.765636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5384.784704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5390.048507] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5399.685088] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5406.099129] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5413.351393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.620310] Lustre: lustre-OST0000: new disk, initializing [ 5413.623986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5413.631108] Lustre: 167949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5413.718950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.151725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5420.162794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5420.299642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5420.701469] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5431.371801] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5436.325547] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5436.498245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5441.166863] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5441.457664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5446.116658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.117526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5446.131526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.155626] Lustre: Skipped 1 previous similar message [ 5449.808070] Lustre: server umount lustre-OST0000 complete [ 5456.355942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.357709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5456.387143] Lustre: Skipped 2 previous similar messages [ 5456.402504] LustreError: Skipped 1 previous similar message [ 5460.393713] Lustre: server umount lustre-MDT0000 complete [ 5461.481547] LustreError: 167007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.506916] LustreError: 167007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5464.528747] LustreError: 165971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845930 with bad export cookie 8612102818445417911 [ 5464.532815] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.534757] LustreError: 165971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5464.962099] Lustre: server umount lustre-MDT0001 complete [ 5475.670657] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:52:19 (1787845939) [ 5492.264720] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5502.185292] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5542.572562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.793463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5542.825821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5542.944104] Lustre: lustre-MDT0000: new disk, initializing [ 5543.035602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.061810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5548.168894] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5561.097323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.324781] Lustre: 172709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5561.688336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5567.252328] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5579.296150] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5586.166702] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.634280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5594.099768] Lustre: lustre-OST0000: new disk, initializing [ 5594.103751] Lustre: Skipped 1 previous similar message [ 5594.127737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5594.141190] Lustre: Skipped 2 previous similar messages [ 5594.155829] Lustre: 174688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5600.755324] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5604.440667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5604.452448] Lustre: Skipped 1 previous similar message [ 5604.460315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5604.598248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5609.642504] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5620.199566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.200555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5620.207347] Lustre: Skipped 1 previous similar message [ 5620.223152] Lustre: Skipped 3 previous similar messages [ 5623.218777] Lustre: server umount lustre-OST0000 complete [ 5628.406121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5628.417299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.470648] Lustre: server umount lustre-MDT0000 complete [ 5635.558254] LustreError: 172719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.592251] LustreError: 172719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5638.324688] LustreError: 172700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846104 with bad export cookie 8612102818445418793 [ 5638.339180] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5638.341788] LustreError: 172700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5638.810093] Lustre: server umount lustre-MDT0001 complete [ 5646.911720] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:55:11 (1787846111) [ 5664.979978] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5674.899961] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5712.944862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.269086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5713.298533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5713.416642] Lustre: lustre-MDT0000: new disk, initializing [ 5713.510972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.516189] Lustre: Skipped 2 previous similar messages [ 5713.546713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5719.982050] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5730.856248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.966742] Lustre: 179013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5731.179498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5736.397411] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5747.187922] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5753.851289] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5761.481470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.963664] Lustre: 180989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5769.159395] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5771.881749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5772.005513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5779.873941] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5787.616961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5787.619662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.647758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5787.654151] Lustre: Skipped 2 previous similar messages [ 5787.664809] Lustre: Skipped 3 previous similar messages [ 5793.487247] Lustre: server umount lustre-OST0000 complete [ 5802.982672] LustreError: 179025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.997579] LustreError: 179025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5803.158591] Lustre: server umount lustre-MDT0000 complete [ 5807.302566] LustreError: 182095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846273 with bad export cookie 8612102818445419878 [ 5807.304585] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5807.318194] LustreError: 182095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5807.970807] Lustre: server umount lustre-MDT0001 complete [ 5819.962147] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5823.217185] Key type lgssc unregistered [ 5823.574394] LNet: 182846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5823.600903] LNetError: 182846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5823.615587] LNet: Removed LNI 192.168.201.116@tcp [ 5824.740427] Key type .llcrypt unregistered [ 5824.752430] Key type ._llcrypt unregistered [ 5849.795111] Key type ._llcrypt registered [ 5849.796918] Key type .llcrypt registered [ 5849.971331] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5851.676591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5851.767574] alg: No test for adler32 (adler32-zlib) [ 5853.030926] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 5853.415674] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5855.135250] Key type lgssc registered [ 5856.578900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5865.670343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5865.696353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.235176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.646287] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5877.730099] LustreError: 184258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.759315] LustreError: 184258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5882.849482] LustreError: 184259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.645155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.350198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5889.946941] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5899.412385] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5907.044470] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5916.468903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.125833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.899365] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5934.132246] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5937.596133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5961.184970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5961.204513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.234561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.234107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.236635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5963.237720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.314058] Lustre: Skipped 1 previous similar message [ 5965.711320] Lustre: server umount lustre-OST0000 complete [ 5971.424958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5971.435727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.446229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5973.476717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.477617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5973.539209] Lustre: Skipped 1 previous similar message [ 5975.659208] Lustre: server umount lustre-MDT0000 complete [ 5978.601157] LustreError: 184254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.628846] LustreError: 184254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5980.965286] LustreError: 184239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846446 with bad export cookie 7559029151537204505 [ 5980.983416] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5980.986982] LustreError: 184239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5981.892116] Lustre: server umount lustre-MDT0001 complete [ 5994.549288] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5997.662917] Key type lgssc unregistered [ 5998.038148] LNet: 187827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5998.045978] LNetError: 187827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5998.069222] LNet: Removed LNI 192.168.201.116@tcp [ 5999.186957] Key type .llcrypt unregistered [ 5999.190591] Key type ._llcrypt unregistered [ 6024.992633] Key type ._llcrypt registered [ 6024.999242] Key type .llcrypt registered [ 6025.137718] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6026.400780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6026.586198] alg: No test for adler32 (adler32-zlib) [ 6027.701364] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 6027.912079] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6029.562401] Key type lgssc registered [ 6030.765636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6039.110867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6039.138638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.753213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.542440] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6050.298810] LustreError: 189238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.318136] LustreError: 189238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6055.401348] LustreError: 189239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.508743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.934129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6062.038660] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6070.312308] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6076.448589] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6085.153867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6085.634144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6090.759540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6092.761414] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6101.612966] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6109.582226] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 6111.200848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6111.211914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.215107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.222992] Lustre: Skipped 1 previous similar message [ 6111.234640] Lustre: Skipped 1 previous similar message [ 6114.673681] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 6114.922909] Lustre: server umount lustre-OST0000 complete [ 6123.277424] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91bfc32db800.ost_server_uuid 50 [ 6125.143223] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6141.920589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.922700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.944970] Lustre: Skipped 1 previous similar message [ 6141.965318] Lustre: Skipped 1 previous similar message [ 6146.497580] Lustre: server umount lustre-MDT0000 complete [ 6147.041230] LustreError: 189233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.064769] LustreError: 189233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6150.366943] LustreError: 189221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846616 with bad export cookie 4981831549449624349 [ 6150.395594] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.400982] LustreError: 189221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6150.857340] Lustre: server umount lustre-MDT0001 complete [ 6160.179328] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:03:43 (1787846623) [ 6167.410230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.057238] LustreError: 193156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.152408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.154029] LustreError: 193157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.374971] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6182.026514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.562903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.388346] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6195.889416] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6202.748504] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6209.794812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6210.449272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.555806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6217.276553] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6228.249382] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6236.881768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.225909] Lustre: lustre-OST0001: new disk, initializing [ 6237.228653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6237.235886] Lustre: 195998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6237.360434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6243.467568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6243.481187] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6243.706457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6245.833459] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6259.536402] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.783293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.787758] LustreError: Skipped 1 previous similar message [ 6263.790445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.799315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.417759] LustreError: 194977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.427841] Lustre: server umount lustre-OST0000 complete [ 6269.431667] LustreError: 194977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6276.394486] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6276.583307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6281.216656] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6281.386288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6286.202641] LustreError: 194980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.222332] LustreError: 194980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6300.138041] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.139810] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6300.142706] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6300.142715] Lustre: Skipped 3 previous similar messages [ 6300.149294] Lustre: Skipped 2 previous similar messages [ 6300.175812] LustreError: Skipped 1 previous similar message [ 6302.414655] Lustre: server umount lustre-OST0001 complete [ 6310.369265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6310.372274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.388396] Lustre: Skipped 1 previous similar message [ 6313.441525] Lustre: server umount lustre-MDT0000 complete [ 6317.664103] LustreError: 193891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846783 with bad export cookie 4981831549449625420 [ 6317.673946] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.680450] LustreError: 193891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6318.010798] Lustre: server umount lustre-MDT0001 complete [ 6344.372926] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:06:48 (1787846808) [ 6352.556926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.912764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6352.966860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6353.154753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6358.006343] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6367.075171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.226384] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6367.261555] Lustre: 199441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6372.083766] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6380.264597] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6386.160484] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6394.462743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.803032] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6403.365824] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6404.830420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6411.780802] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6419.220916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6419.311891] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6419.593438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6419.603635] Lustre: Skipped 2 previous similar messages [ 6426.619094] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6434.528494] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6440.324925] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 6440.333578] Lustre: Skipped 3 previous similar messages [ 6440.417751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.430995] Lustre: Skipped 1 previous similar message [ 6443.116452] Lustre: server umount lustre-OST0000 complete [ 6445.422840] LustreError: 201293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.447723] LustreError: 201293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6460.896289] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6460.900922] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.910740] LustreError: Skipped 1 previous similar message [ 6460.933341] Lustre: Skipped 1 previous similar message [ 6467.051829] Lustre: server umount lustre-OST0001 complete [ 6476.257756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6476.261637] Lustre: Skipped 9 previous similar messages [ 6477.321403] Lustre: server umount lustre-MDT0000 complete [ 6482.083286] LustreError: 199433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846948 with bad export cookie 4981831549449626820 [ 6482.085828] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.103760] LustreError: 199433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6482.528495] Lustre: server umount lustre-MDT0001 complete [ 6510.503440] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:09:34 (1787846974) [ 6528.235468] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6538.341791] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6575.266809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.534888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6575.555389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6575.634973] Lustre: lustre-MDT0000: new disk, initializing [ 6575.701253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.724217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6581.694611] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6595.097576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.348909] Lustre: 207539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6595.427073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6595.436082] Lustre: Skipped 1 previous similar message [ 6595.545886] Lustre: lustre-MDT0001: new disk, initializing [ 6595.792274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6595.832432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6602.642872] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6613.773892] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6620.942406] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6629.976132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6630.176206] Lustre: lustre-OST0000: new disk, initializing [ 6630.177838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6630.180452] Lustre: 209510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6638.140294] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6638.196301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6638.210561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6638.356642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6650.337649] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6655.977632] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6656.238678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6661.042763] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6661.207821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.160100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6664.163355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.166347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.177911] Lustre: Skipped 3 previous similar messages [ 6664.189708] Lustre: Skipped 1 previous similar message [ 6669.881075] Lustre: server umount lustre-OST0000 complete [ 6680.853895] Lustre: server umount lustre-MDT0000 complete [ 6684.641590] LustreError: 207552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.673914] LustreError: 207552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6685.672399] LustreError: 207532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847151 with bad export cookie 4981831549449628388 [ 6685.689822] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.913714] Lustre: server umount lustre-MDT0001 complete [ 6693.107380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.405554] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6708.051382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.653288] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6720.840284] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6726.998195] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6734.771966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6741.381237] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6747.797294] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6752.125640] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6752.340451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6757.664622] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6757.843072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6759.915646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.922277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6759.939047] Lustre: Skipped 2 previous similar messages [ 6759.948237] LustreError: Skipped 2 previous similar messages [ 6762.263340] Lustre: server umount lustre-OST0000 complete [ 6768.989962] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6774.012680] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6805.138689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6813.966744] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6821.139700] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.255640] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6825.974601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6825.988061] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6825.994817] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6826.016868] Lustre: Skipped 1 previous similar message [ 6831.073446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6831.090368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6831.093715] Lustre: Skipped 8 previous similar messages [ 6834.084943] Lustre: server umount lustre-OST0000 complete [ 6845.361191] Lustre: server umount lustre-MDT0000 complete [ 6846.441881] LustreError: 211729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.453768] LustreError: 211729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6848.653958] LustreError: 213573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847314 with bad export cookie 4981831549449629270 [ 6848.655079] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6848.661816] LustreError: 213573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6848.949254] Lustre: server umount lustre-MDT0001 complete [ 6857.125212] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:15:21 (1787847321) [ 6863.009068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.625455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.642387] Lustre: Skipped 6 previous similar messages [ 6868.216791] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6876.931271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.335893] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6890.441982] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6895.486798] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6901.727698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6908.249383] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6915.805920] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6919.359493] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6919.546556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6923.392542] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6923.582795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6929.854051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.099908] Lustre: lustre-OST0001: new disk, initializing [ 6930.110310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6930.126046] Lustre: 221066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6936.519969] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6936.670081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6936.679975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6936.788589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6946.820420] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6951.316235] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6951.547365] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6955.828470] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6956.018650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6962.660124] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6962.675273] Lustre: Skipped 5 previous similar messages [ 6962.680144] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6962.695700] LustreError: Skipped 1 previous similar message [ 6972.383546] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6972.665303] Lustre: server umount lustre-OST0001 complete [ 6978.595473] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6978.782331] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6982.804329] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6983.003569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7013.303239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.985727] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7018.999533] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7018.999687] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7019.021304] Lustre: Skipped 1 previous similar message [ 7019.790637] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7028.285044] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7030.524921] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7039.333650] Lustre: server umount lustre-OST0001 complete [ 7063.690455] Lustre: server umount lustre-OST0000 complete [ 7073.273906] Lustre: server umount lustre-MDT0000 complete [ 7076.848928] LustreError: 217677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847542 with bad export cookie 4981831549449630236 [ 7076.861298] LustreError: 217677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7076.864911] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7091.167818] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7091.445812] Lustre: server umount lustre-MDT0001 complete [ 7112.871123] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:19:37 (1787847577) [ 7118.580655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.939574] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7118.971216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.192642] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7131.184675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.241348] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7131.263489] Lustre: 226311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7135.616112] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7143.911896] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7149.320553] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7155.937773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7156.098691] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7161.947471] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7169.603347] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7178.177451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.305635] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7184.851334] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7191.269686] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7192.738198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7196.805285] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7197.035417] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7201.293432] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7201.464948] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7206.312641] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91bfc9389800.ost_server_uuid 50 [ 7207.458894] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91bfc9389800.ost_server_uuid in IDLE state after 0 sec [ 7208.339419] Lustre: Permanently deactivating lustre-OST0001 [ 7208.352511] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7211.041909] Lustre: Permanently reactivating lustre-OST0001 [ 7218.144214] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7218.152143] LustreError: Skipped 3 previous similar messages [ 7218.153823] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7218.170348] Lustre: Skipped 13 previous similar messages [ 7220.699780] Lustre: server umount lustre-OST0001 complete [ 7223.271744] LustreError: 228157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7223.290925] LustreError: 228157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7228.388953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.408810] Lustre: Skipped 9 previous similar messages [ 7239.135152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7239.319825] Lustre: server umount lustre-OST0000 complete [ 7248.747083] Lustre: server umount lustre-MDT0000 complete [ 7252.105454] LustreError: 226303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847718 with bad export cookie 4981831549449631524 [ 7252.106862] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7252.118679] LustreError: 226303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7252.429271] Lustre: server umount lustre-MDT0001 complete [ 7273.676762] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:22:17 (1787847737) [ 7280.875070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.305554] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7281.375888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7281.404940] Lustre: Skipped 5 previous similar messages [ 7286.663263] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7295.320358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7295.439633] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7295.469686] Lustre: 233001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7300.124650] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7308.583100] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7314.644497] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7323.811932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.059431] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7324.088780] Lustre: Permanently deactivating lustre-OST0000 [ 7324.108547] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7330.887859] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7332.193420] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7339.789471] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7346.886813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.006738] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7353.089337] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7353.954450] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7362.355990] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7364.373827] Lustre: Permanently reactivating lustre-OST0000 [ 7364.387246] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7364.390922] Lustre: Skipped 2 previous similar messages [ 7374.511432] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7374.512251] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000c11a2282, cur 1787847840 deadline 1787847826 last 1787847796 [ 7374.521188] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7374.591740] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7374.686477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7378.965777] LustreError: 236628:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007bd0e8d3 x1874692924755072/t0(0) o104->lustre-OST0001@192.168.201.16@tcp:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 7396.319166] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7396.404371] Lustre: server umount lustre-OST0001 complete [ 7406.630690] Lustre: server umount lustre-OST0000 complete [ 7417.641072] Lustre: server umount lustre-MDT0000 complete [ 7421.858460] LustreError: 232993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847887 with bad export cookie 4981831549449633169 [ 7421.860947] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.881068] LustreError: 232993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7422.292646] Lustre: server umount lustre-MDT0001 complete [ 7434.957504] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7438.332622] Key type lgssc unregistered [ 7438.622341] LNet: 237883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7438.628485] LNetError: 237883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7439.661499] LNet: Removed LNI 192.168.201.116@tcp [ 7440.872601] Key type .llcrypt unregistered [ 7440.874926] Key type ._llcrypt unregistered [ 7468.141094] Key type ._llcrypt registered [ 7468.142524] Key type .llcrypt registered [ 7468.321596] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7469.670928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7469.705310] alg: No test for adler32 (adler32-zlib) [ 7470.904665] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 7471.287605] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7473.073518] Key type lgssc registered [ 7474.676778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7486.597498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7486.646357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.314093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.024529] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7498.726418] LustreError: 239348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.762358] LustreError: 239348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7503.850347] LustreError: 239347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.090463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7504.316559] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7504.361186] Lustre: 239337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7504.384112] Lustre: Permanently deactivating lustre-MDT0001 [ 7504.431336] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7504.951103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7511.865305] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7512.610381] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7521.312955] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7527.388318] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7534.722693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7535.220542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.712814] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7549.554766] LustreError: 241189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.591433] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7554.149437] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7555.063380] LustreError: 241187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.154063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7562.015272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.207169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7567.356372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7567.378053] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7569.430869] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7577.693445] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7580.977895] Lustre: Permanently reactivating lustre-MDT0001 [ 7581.008086] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7581.011068] Lustre: Skipped 2 previous similar messages [ 7586.187391] Lustre: lustre-MDT0001: Client c1600250-be2d-4703-bfa7-837041b87493 (at 192.168.201.16@tcp) reconnecting [ 7587.041531] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.058931] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7587.073211] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7587.092514] LustreError: 242913:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7587.097951] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7587.110201] LustreError: 242913:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7587.110351] LustreError: 242913:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.116@tcp: cfg command failed: rc = -108 [ 7587.110375] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7587.110375] [ 7587.111521] LustreError: 239340:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7593.209935] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7595.066989] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7602.192308] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7602.420987] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7608.236889] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7608.342665] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7610.554956] Lustre: Permanently deactivating lustre-MDT0001 [ 7610.579192] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7610.587626] Lustre: Skipped 2 previous similar messages [ 7620.847074] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7632.864785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7632.874230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.883225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7633.888487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7633.891533] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.899595] Lustre: Skipped 2 previous similar messages [ 7633.919280] Lustre: Skipped 2 previous similar messages [ 7638.068980] Lustre: server umount lustre-MDT0000 complete [ 7639.021315] LustreError: 239347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7639.053180] LustreError: 239347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7642.829762] LustreError: 239327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848108 with bad export cookie 14682391178641924659 [ 7642.833448] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.846430] LustreError: 239327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7643.351043] Lustre: server umount lustre-MDT0001 complete [ 7650.093706] Lustre: server umount lustre-OST0000 complete [ 7664.090645] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:28:47 (1787848127) [ 7675.409856] Lustre: server umount lustre-OST0001 complete [ 7682.385384] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7694.940879] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7737.131059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.511704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7737.522991] Lustre: Skipped 2 previous similar messages [ 7737.544469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7737.658236] Lustre: lustre-MDT0000: new disk, initializing [ 7737.826173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7737.861188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7743.488834] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7754.931474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.138690] Lustre: 247661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7755.253769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7755.264079] Lustre: Skipped 1 previous similar message [ 7755.335796] Lustre: lustre-MDT0001: new disk, initializing [ 7755.426172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7755.465461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7755.483971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7761.222180] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7773.445339] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7781.886429] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7789.920811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7790.349254] Lustre: lustre-OST0000: new disk, initializing [ 7790.356604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7790.367681] Lustre: 249631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7790.499169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.865607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7796.875444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7797.038035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7797.184710] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7807.115684] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7809.181077] LustreError: 249225:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7819.198253] LustreError: 249225:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7819.232762] LustreError: 247668:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7819.615737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.780161] Lustre: lustre-OST0001: new disk, initializing [ 7819.783907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7819.789614] Lustre: 250743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7828.424942] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7829.174866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7829.188676] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7829.284541] LustreError: 247668:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7829.311531] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7841.793853] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7844.333930] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7849.963467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7849.982668] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7850.008042] LustreError: Skipped 1 previous similar message [ 7850.010298] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7850.038763] Lustre: Skipped 1 previous similar message [ 7850.074552] Lustre: Skipped 2 previous similar messages [ 7853.127371] Lustre: server umount lustre-OST0001 complete [ 7855.084317] LustreError: 249616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.141366] LustreError: 249616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7874.025229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7874.028285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.044929] LustreError: Skipped 1 previous similar message [ 7874.050900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7874.070616] Lustre: Skipped 1 previous similar message [ 7874.097557] Lustre: Skipped 1 previous similar message [ 7875.565501] LustreError: 250752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.600331] LustreError: 250752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7880.687187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7880.703362] Lustre: Skipped 3 previous similar messages [ 7885.279377] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7885.725402] Lustre: server umount lustre-OST0000 complete [ 7890.914270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.918932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7890.940738] Lustre: Skipped 1 previous similar message [ 7895.991032] Lustre: server umount lustre-MDT0000 complete [ 7900.192293] LustreError: 247653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848366 with bad export cookie 14682391178641927466 [ 7900.194052] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7900.203798] LustreError: 247653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7906.803167] Lustre: server umount lustre-MDT0001 complete [ 7921.084726] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7924.342888] Key type lgssc unregistered [ 7924.702168] LNet: 252782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7924.706956] LNetError: 252782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.724517] LNet: Removed LNI 192.168.201.116@tcp [ 7925.642794] Key type .llcrypt unregistered [ 7925.645962] Key type ._llcrypt unregistered [ 7973.451210] Key type ._llcrypt registered [ 7973.453481] Key type .llcrypt registered [ 7973.619373] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7975.256262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7975.345052] alg: No test for adler32 (adler32-zlib) [ 7976.425571] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 7976.756936] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7978.513053] Key type lgssc registered [ 7979.618270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7988.799985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7988.815033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.087806] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7990.112363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7990.239120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.471971] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8004.493757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.612457] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8004.664102] Lustre: 254988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8005.023734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8010.067402] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8017.880215] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8025.011653] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8032.995961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.255197] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8033.401430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.899485] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8042.730497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8048.699164] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8109.537447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8109.554088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.556356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8109.575024] Lustre: Skipped 1 previous similar message [ 8109.587709] Lustre: Skipped 1 previous similar message [ 8112.007982] Lustre: server umount lustre-OST0000 complete [ 8117.298551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8132.833443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8133.582697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8139.622378] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8147.995548] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8149.993949] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8150.018056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8150.022100] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8150.058546] Lustre: Skipped 1 previous similar message [ 8190.944325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8190.947371] LustreError: Skipped 1 previous similar message [ 8190.950982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.960727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8190.986439] Lustre: Skipped 1 previous similar message [ 8193.898635] Lustre: server umount lustre-OST0000 complete [ 8199.648254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8199.652733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.681213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8199.687308] Lustre: Skipped 1 previous similar message [ 8204.693640] Lustre: server umount lustre-MDT0000 complete [ 8208.568895] LustreError: 256852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848674 with bad export cookie 7552088062358177342 [ 8208.578419] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8208.595924] LustreError: 256852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8208.995541] Lustre: server umount lustre-MDT0001 complete [ 8220.210557] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8222.938114] Key type lgssc unregistered [ 8223.293552] LNet: 260415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8223.306836] LNetError: 260415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8223.327297] LNet: Removed LNI 192.168.201.116@tcp [ 8224.335589] Key type .llcrypt unregistered [ 8224.338260] Key type ._llcrypt unregistered [ 8251.084080] Key type ._llcrypt registered [ 8251.086844] Key type .llcrypt registered [ 8251.171126] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8252.767750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8252.808529] alg: No test for adler32 (adler32-zlib) [ 8254.002670] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 8254.308754] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8255.977145] Key type lgssc registered [ 8257.610613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8267.043344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8267.066553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.473726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8272.730146] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8279.024316] LustreError: 261829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.046277] LustreError: 261829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8281.378907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.755445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8286.802661] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8294.191858] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8299.445061] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8305.623439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8305.974439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.546137] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8314.176590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8318.331826] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8343.023976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.043911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.071706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8345.056200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8345.069449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.081697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8345.090821] Lustre: Skipped 1 previous similar message [ 8348.706567] Lustre: server umount lustre-OST0000 complete [ 8353.249665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8353.257861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.269114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.301899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.312479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.357398] Lustre: Skipped 1 previous similar message [ 8359.598544] Lustre: server umount lustre-MDT0000 complete [ 8360.419960] LustreError: 261829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8360.440665] LustreError: 261829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8364.423473] LustreError: 261810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848830 with bad export cookie 796050608220107820 [ 8364.430541] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8364.441551] LustreError: 261810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8365.001282] Lustre: server umount lustre-MDT0001 complete [ 8383.280286] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8386.268243] Key type lgssc unregistered [ 8386.508380] LNet: 266155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8386.512796] LNetError: 266155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8386.541581] LNet: Removed LNI 192.168.201.116@tcp [ 8387.622266] Key type .llcrypt unregistered [ 8387.630171] Key type ._llcrypt unregistered [ 8410.933278] Key type ._llcrypt registered [ 8410.946923] Key type .llcrypt registered [ 8411.092537] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8412.574834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8412.734779] alg: No test for adler32 (adler32-zlib) [ 8413.712819] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 8413.989710] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8415.679335] Key type lgssc registered [ 8416.903248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8425.493635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8425.547806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.260952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.780848] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8437.744290] LustreError: 267374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.759818] LustreError: 267374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8442.850298] LustreError: 267375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.470638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8443.835874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8448.397942] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8457.449466] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8463.858963] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8471.539988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.887808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.949807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8478.842033] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8486.936749] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8498.657558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8498.665170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.679761] LustreError: Skipped 1 previous similar message [ 8498.680019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8498.716296] Lustre: Skipped 1 previous similar message [ 8503.781239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8503.803044] Lustre: Skipped 2 previous similar messages [ 8504.542184] Lustre: server umount lustre-OST0000 complete [ 8508.897283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.904901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8508.920052] Lustre: Skipped 1 previous similar message [ 8508.946219] Lustre: Skipped 1 previous similar message [ 8514.017761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.027340] Lustre: Skipped 1 previous similar message [ 8514.787260] Lustre: server umount lustre-MDT0000 complete [ 8518.702173] LustreError: 267356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848984 with bad export cookie 8913954856302697344 [ 8518.712844] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8519.099360] Lustre: server umount lustre-MDT0001 complete [ 8531.514438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8535.080730] Key type lgssc unregistered [ 8535.343633] LNet: 271052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8535.356119] LNetError: 271052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8535.371534] LNet: Removed LNI 192.168.201.116@tcp [ 8536.179177] Key type .llcrypt unregistered [ 8536.185486] Key type ._llcrypt unregistered [ 8563.242225] Key type ._llcrypt registered [ 8563.248516] Key type .llcrypt registered [ 8563.339623] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8564.858052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8564.940413] alg: No test for adler32 (adler32-zlib) [ 8566.323369] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 8566.672505] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8568.415178] Key type lgssc registered [ 8569.904703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8579.611412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8579.632520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.424511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.458531] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8591.841904] LustreError: 272467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.873261] LustreError: 272467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8595.121558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.651305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.631992] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8609.393932] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8616.410609] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8624.129570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.464675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.598038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8631.484425] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8639.931340] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8661.498939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.501250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8661.525838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8661.549644] Lustre: Skipped 1 previous similar message [ 8666.595419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8667.843807] Lustre: server umount lustre-OST0000 complete [ 8671.718179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.733085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.735185] Lustre: Skipped 2 previous similar messages [ 8671.753025] Lustre: Skipped 2 previous similar messages [ 8676.848314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8677.940640] Lustre: server umount lustre-MDT0000 complete [ 8681.194124] LustreError: 272449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849147 with bad export cookie 2818836156216135120 [ 8681.208977] LustreError: 272449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8681.218676] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8681.516848] Lustre: server umount lustre-MDT0001 complete [ 8692.365571] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8695.241987] Key type lgssc unregistered [ 8695.514595] LNet: 276775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8695.531899] LNetError: 276775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8696.551829] LNet: Removed LNI 192.168.201.116@tcp [ 8697.474415] Key type .llcrypt unregistered [ 8697.479315] Key type ._llcrypt unregistered [ 8713.456259] Key type ._llcrypt registered [ 8713.461127] Key type .llcrypt registered [ 8713.625792] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8714.703194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8714.810537] alg: No test for adler32 (adler32-zlib) [ 8715.994494] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 8716.325168] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8717.999174] Key type lgssc registered [ 8719.223761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8726.429341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8726.454495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.041410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.914949] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8738.276039] LustreError: 278008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.298776] LustreError: 278008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8741.413530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.679501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8746.153478] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8754.590843] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8761.779256] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8769.197153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8769.624779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.770335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8776.965818] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8785.349446] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8799.712690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8799.719915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.738855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8801.760497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8801.776758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8801.787741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8804.610888] Lustre: server umount lustre-OST0000 complete [ 8808.420038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8808.425068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8808.437861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8808.446037] Lustre: Skipped 1 previous similar message [ 8812.000673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8812.002356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8814.574740] Lustre: server umount lustre-MDT0000 complete [ 8817.120848] LustreError: 278019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8817.135967] LustreError: 278019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8818.360931] LustreError: 277974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849284 with bad export cookie 11097225446800244725 [ 8818.370623] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8818.377994] LustreError: 277974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8818.811173] Lustre: server umount lustre-MDT0001 complete [ 8831.746618] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8834.575979] Key type lgssc unregistered [ 8834.852790] LNet: 281708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8834.864351] LNetError: 281708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8835.879942] LNet: Removed LNI 192.168.201.116@tcp [ 8836.949452] Key type .llcrypt unregistered [ 8836.951068] Key type ._llcrypt unregistered [ 8849.019740] Key type ._llcrypt registered [ 8849.021792] Key type .llcrypt registered [ 8849.174858] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8869.848687] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8889.975826] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8891.854141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8891.947118] alg: No test for adler32 (adler32-zlib) [ 8893.105328] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 8893.518314] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8895.343161] Key type lgssc registered [ 8896.993826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8933.494173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8933.545559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8934.844927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8934.905112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8935.052677] Lustre: lustre-MDT0000: new disk, initializing [ 8935.126516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.157421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8940.229661] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8952.401831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8952.598421] Lustre: 285262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8952.732732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8952.740391] Lustre: Skipped 1 previous similar message [ 8952.916794] Lustre: lustre-MDT0001: new disk, initializing [ 8953.050222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8953.142453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8953.160993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8957.291489] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8967.640588] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8974.251036] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8981.415264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8981.788549] Lustre: lustre-OST0000: new disk, initializing [ 8981.794744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8981.803793] Lustre: 287230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8981.996602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.665289] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8990.845098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8990.864716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8991.032406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8998.635689] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9004.100456] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9004.308701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9009.607560] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9009.767989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9016.807604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.838891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9016.851251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9016.864556] Lustre: Skipped 1 previous similar message [ 9021.923381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9021.931978] Lustre: Skipped 1 previous similar message [ 9026.015340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9026.288889] Lustre: server umount lustre-OST0000 complete [ 9030.121427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9030.134863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.160426] Lustre: Skipped 1 previous similar message [ 9030.187188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9032.166183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9032.209046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9032.224121] Lustre: Skipped 1 previous similar message [ 9036.248960] Lustre: server umount lustre-MDT0000 complete [ 9042.404519] LustreError: 285274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.469873] LustreError: 285274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9043.885899] LustreError: 285252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849509 with bad export cookie 6121921070356537672 [ 9043.910770] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9043.923037] LustreError: 285252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9045.155438] Lustre: server umount lustre-MDT0001 complete [ 9063.164974] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:52:06 (1787849526) [ 9075.670850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.467583] LustreError: 289646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.576328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.825533] LustreError: 289647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.052333] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9086.955416] LustreError: 289646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.067697] LustreError: 289647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.900597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.469174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9098.437746] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9106.419442] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9112.126559] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9118.507202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.738607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9125.308622] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9134.912711] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.360042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9147.363987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.372968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9153.523684] Lustre: server umount lustre-OST0000 complete [ 9158.624187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.627901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9158.640166] Lustre: Skipped 2 previous similar messages [ 9158.657428] Lustre: Skipped 3 previous similar messages [ 9163.436549] Lustre: server umount lustre-MDT0000 complete [ 9163.745738] LustreError: 289647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9163.761439] LustreError: 289647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9166.642816] LustreError: 289628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849632 with bad export cookie 6121921070356538561 [ 9166.656251] LustreError: 289628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9166.678122] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9166.955373] Lustre: server umount lustre-MDT0001 complete [ 9178.439812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9181.554508] Key type lgssc unregistered [ 9181.852630] LNet: 293192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9181.868711] LNetError: 293192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9181.901604] LNet: Removed LNI 192.168.201.116@tcp [ 9182.824460] Key type .llcrypt unregistered [ 9182.832947] Key type ._llcrypt unregistered [ 9210.833501] Key type ._llcrypt registered [ 9210.835739] Key type .llcrypt registered [ 9210.958615] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9227.489638] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9228.808895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.941157] alg: No test for adler32 (adler32-zlib) [ 9230.111150] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 9230.548923] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 9232.271252] Key type lgssc registered [ 9233.909381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9280.504787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9280.528688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.882522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9281.988058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9282.115493] Lustre: lustre-MDT0000: new disk, initializing [ 9282.230509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9282.250703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9287.191159] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9297.858037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.027947] Lustre: 297016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9298.098222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9298.105722] Lustre: Skipped 1 previous similar message [ 9298.168723] Lustre: lustre-MDT0001: new disk, initializing [ 9298.262188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9298.314882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9298.331878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9302.722582] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9311.145964] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9317.330535] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9324.087864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9324.437736] Lustre: lustre-OST2710: new disk, initializing [ 9324.441944] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9324.449283] Lustre: 298981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9324.541682] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9331.039774] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9333.848209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9333.864408] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9333.946200] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9340.667446] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9347.298934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9347.438265] Lustre: lustre-OST03e8: new disk, initializing [ 9347.441706] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9347.446961] Lustre: 300044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9347.530240] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9354.015183] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9356.401775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9356.416388] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9356.500523] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9360.787702] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9366.967686] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9367.219725] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9371.612195] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9371.891443] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9376.910660] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9377.125488] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9382.657993] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9383.083820] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9387.501292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9387.517329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9387.520689] Lustre: Skipped 3 previous similar messages [ 9387.548299] Lustre: Skipped 3 previous similar messages [ 9390.027548] Lustre: server umount lustre-MDT0000 complete [ 9394.938282] LustreError: 297006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849860 with bad export cookie 7485740774062918448 [ 9394.946223] LustreError: 297006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9394.957162] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9395.480269] Lustre: server umount lustre-MDT0001 complete [ 9411.557468] Lustre: server umount lustre-OST2710 complete [ 9413.151754] Lustre: 294812:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849863/real 1787849863] req@00000000c43056fc x1874696282899072/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9413.209178] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9415.135238] Lustre: 294810:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849865/real 1787849865] req@000000005424914e x1874696282899328/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9417.623499] Lustre: server umount lustre-OST03e8 complete [ 9442.863695] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 9453.901530] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9489.422670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.671983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9489.693823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9489.760050] Lustre: lustre-MDT0000: new disk, initializing [ 9489.837557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9489.866379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9494.119337] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9504.236817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.403331] Lustre: 305295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9504.524473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9504.537904] Lustre: Skipped 1 previous similar message [ 9504.643101] Lustre: lustre-MDT0001: new disk, initializing [ 9504.768097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9504.809849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9504.820019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9509.479713] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9519.246703] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9524.775864] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9532.404716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9532.622034] Lustre: lustre-OST0000: new disk, initializing [ 9532.634078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9532.639421] Lustre: 307265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9532.710534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.265888] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9542.306988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9542.332631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9542.437508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9547.538313] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9551.467657] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9551.668983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9555.507367] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9555.708737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9558.001977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.008928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.025399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9558.029449] Lustre: Skipped 1 previous similar message [ 9563.118337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.122769] Lustre: Skipped 2 previous similar messages [ 9563.616573] Lustre: server umount lustre-OST0000 complete [ 9568.224773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.256913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.271193] Lustre: Skipped 1 previous similar message [ 9573.351637] LustreError: 305301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9573.363486] LustreError: 305301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9573.433702] Lustre: server umount lustre-MDT0000 complete [ 9577.060423] LustreError: 305288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850043 with bad export cookie 7485740774062920128 [ 9577.064576] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.073498] LustreError: 305288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9577.387456] Lustre: server umount lustre-MDT0001 complete [ 9587.257355] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:00:51 (1787850051) [ 9588.743533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9590.827472] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:00:54 (1787850054) [ 9594.068533] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9621.493200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.666847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9621.682606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9621.830526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9626.829869] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9635.038708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.112885] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9635.144492] Lustre: 311131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9640.190892] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9662.261191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9662.433788] Lustre: Denying initial registration attempt from nid 192.168.201.116@tcp, specified as failover [ 9662.448996] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9662.457343] LustreError: 312586:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9662.474671] LustreError: 312586:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9662.480674] LustreError: 312586:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9662.575842] Lustre: server umount lustre-OST0000 complete [ 9662.579814] LustreError: 312586:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9671.137704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9671.151559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9671.169363] Lustre: Skipped 1 previous similar message [ 9671.173662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9675.348985] Lustre: server umount lustre-MDT0000 complete [ 9678.310945] LustreError: 311137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.327185] LustreError: 311137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9678.892691] LustreError: 311125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850144 with bad export cookie 7485740774062921010 [ 9678.894766] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9678.903387] LustreError: 311125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9679.226197] Lustre: server umount lustre-MDT0001 complete [ 9689.819628] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9692.610409] Key type lgssc unregistered [ 9692.926828] LNet: 313707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9692.938716] LNetError: 313707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9692.959824] LNet: Removed LNI 192.168.201.116@tcp [ 9693.991174] Key type .llcrypt unregistered [ 9693.995201] Key type ._llcrypt unregistered [ 9704.884528] Key type ._llcrypt registered [ 9704.889690] Key type .llcrypt registered [ 9705.028327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9706.491790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9706.559133] alg: No test for adler32 (adler32-zlib) [ 9707.720539] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 9708.038259] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 9709.791206] Key type lgssc registered [ 9711.316552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9763.153248] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9771.104721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9771.117056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.346992] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9772.378133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9772.539810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.461451] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9786.402870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.575829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9786.635943] Lustre: 316469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9786.927416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9791.687408] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9800.811705] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9808.034891] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9830.066200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9830.317281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9830.655945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.733680] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9844.785253] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9850.340788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9850.341545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9850.348778] LustreError: Skipped 1 previous similar message [ 9850.353189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9850.379858] Lustre: Skipped 1 previous similar message [ 9853.388479] Lustre: server umount lustre-OST0000 complete [ 9858.528483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9858.541844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9858.572219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9858.580651] Lustre: Skipped 1 previous similar message [ 9860.577128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9860.584475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.056820] Lustre: server umount lustre-MDT0000 complete [ 9867.900266] LustreError: 316463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850333 with bad export cookie 17839665667671835217 [ 9867.908425] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9867.912223] LustreError: 316463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9868.301385] Lustre: server umount lustre-MDT0001 complete [ 9880.775678] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 9884.630912] Key type lgssc unregistered [ 9885.063586] LNet: 320046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9885.081747] LNetError: 320046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9885.103964] LNet: Removed LNI 192.168.201.116@tcp [ 9886.379876] Key type .llcrypt unregistered [ 9886.385139] Key type ._llcrypt unregistered [ 9914.277895] Key type ._llcrypt registered [ 9914.279297] Key type .llcrypt registered [ 9914.439468] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 9915.411679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9915.424458] alg: No test for adler32 (adler32-zlib) [ 9916.633329] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 9916.803110] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 9918.463175] Key type lgssc registered [ 9920.021539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9928.289923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9928.297109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.702556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.617904] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9939.943929] LustreError: 321462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9939.969085] LustreError: 321462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9944.037142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.464879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9949.719716] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9957.496958] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9962.751787] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9971.581608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9971.997468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9977.974334] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 9985.551387] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9998.756035] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 9999.023669] Lustre: server umount lustre-MDT0000 complete [ 9999.841405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.845219] LustreError: 321458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.863185] Lustre: Skipped 1 previous similar message [ 9999.905106] LustreError: 321458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10002.872067] LustreError: 321443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850468 with bad export cookie 9404946296737856838 [10002.887961] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10003.824889] LustreError: 321457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10003.851894] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [10004.961018] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10004.969722] Lustre: Skipped 1 previous similar message [10004.986462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10007.976493] LustreError: 321458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10007.980434] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10007.998886] LustreError: 321458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10012.064549] LustreError: 321457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.064596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10012.089105] LustreError: 321457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10012.104434] Lustre: Skipped 1 previous similar message [10017.260624] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10017.607433] Lustre: server umount lustre-MDT0001 complete [10023.220960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10030.791335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.401256] LustreError: 323293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.416263] LustreError: 323293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10048.505844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10048.557915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10053.166062] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10061.647585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.973263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10067.579660] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10072.564556] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10075.367908] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10077.518749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10084.757630] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10086.264520] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10093.044946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10093.049780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10093.059680] Lustre: Skipped 1 previous similar message [10093.078700] Lustre: Skipped 3 previous similar messages [10107.359962] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10107.695173] Lustre: server umount lustre-OST0000 complete [10112.992910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10113.005233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10113.031283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10113.047195] Lustre: Skipped 4 previous similar messages [10118.008148] Lustre: server umount lustre-MDT0000 complete [10118.632189] LustreError: 325079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.657190] LustreError: 325079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10122.584717] LustreError: 321443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850588 with bad export cookie 9404946296737863978 [10122.593675] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10122.599123] LustreError: 321443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10123.165524] Lustre: server umount lustre-MDT0001 complete [10136.197859] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10139.017280] Key type lgssc unregistered [10139.266111] LNet: 327752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10139.277566] LNetError: 327752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10139.304608] LNet: Removed LNI 192.168.201.116@tcp [10140.070969] Key type .llcrypt unregistered [10140.074965] Key type ._llcrypt unregistered [10166.159982] Key type ._llcrypt registered [10166.162347] Key type .llcrypt registered [10166.246176] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10167.071256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10167.099844] alg: No test for adler32 (adler32-zlib) [10168.382347] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [10168.770629] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [10170.481786] Key type lgssc registered [10172.122976] Lustre: Echo OBD driver; http://www.lustre.org/ [10180.252747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10180.269843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.768060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10187.216111] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10191.345866] LustreError: 329173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.389632] LustreError: 329173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10196.186773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.452424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10196.662237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10201.603329] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10209.144394] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10214.721722] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10221.536490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.539391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10221.550120] Lustre: Skipped 1 previous similar message [10221.564821] Lustre: Skipped 1 previous similar message [10222.562386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10222.571479] Lustre: Skipped 1 previous similar message [10227.682883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10227.690781] Lustre: Skipped 1 previous similar message [10232.292476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10232.714540] Lustre: server umount lustre-MDT0000 complete [10232.808570] LustreError: 329167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.822915] LustreError: 329167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10236.304723] LustreError: 329151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850702 with bad export cookie 6843647935265336364 [10236.316861] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10236.320162] LustreError: 329151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10236.628499] Lustre: server umount lustre-MDT0001 complete [10252.183946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.560618] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10252.610140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10252.931971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10258.027249] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10267.719882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10267.885796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10267.965186] Lustre: 331761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10268.318144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10273.243953] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10281.215259] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10286.864771] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10296.624493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10296.972719] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10296.980454] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10296.990327] LustreError: 333645:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10296.997502] LustreError: 333645:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10297.003850] LustreError: 333645:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10297.105471] Lustre: server umount lustre-OST0000 complete [10297.111988] LustreError: 333645:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10311.333034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10311.542674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10311.896426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10319.185219] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10321.184527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10328.310716] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10336.580932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.756427] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10336.908749] Lustre: lustre-OST0001: new disk, initializing [10336.912220] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10336.919811] Lustre: 335188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10336.992881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10345.056218] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10347.124966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10347.136913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10347.256280] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10352.138810] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10357.741962] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10357.748700] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10357.752144] LustreError: Skipped 1 previous similar message [10357.775215] Lustre: Skipped 1 previous similar message [10357.779805] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10357.785545] Lustre: Skipped 1 previous similar message [10362.008040] Lustre: server umount lustre-OST0001 complete [10362.867993] LustreError: 334113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.988966] LustreError: 334114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.026537] LustreError: 334114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10372.577425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10372.588299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.594887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.597565] Lustre: Skipped 1 previous similar message [10372.608577] Lustre: Skipped 1 previous similar message [10373.104731] LustreError: 334114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.115753] LustreError: 334114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10374.183380] Lustre: server umount lustre-OST0000 complete [10383.329176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10388.450242] LustreError: 331771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.603213] Lustre: server umount lustre-MDT0000 complete [10392.969356] LustreError: 332503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850858 with bad export cookie 6843647935265336917 [10392.983040] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10392.984353] LustreError: 332503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10393.472525] Lustre: server umount lustre-MDT0001 complete [10406.097390] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [10409.338850] Key type lgssc unregistered [10409.832678] LNet: 337145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10409.839711] LNetError: 337145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10409.895782] LNet: Removed LNI 192.168.201.116@tcp [10411.073695] Key type .llcrypt unregistered [10411.076059] Key type ._llcrypt unregistered [10487.042612] Key type ._llcrypt registered [10487.049317] Key type .llcrypt registered [10487.155802] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [10505.323346] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [10506.871944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10506.949159] alg: No test for adler32 (adler32-zlib) [10508.068353] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [10508.338717] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [10510.042513] Key type lgssc registered [10511.042679] Lustre: Echo OBD driver; http://www.lustre.org/ [10548.717648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10548.781804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.026036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10550.081740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10550.251487] Lustre: lustre-MDT0000: new disk, initializing [10550.366269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10550.396226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10555.420851] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10566.671382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10566.807217] Lustre: 342077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10566.848539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10566.868070] Lustre: Skipped 1 previous similar message [10566.958154] Lustre: lustre-MDT0001: new disk, initializing [10567.071449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10567.114959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10567.130228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10572.058776] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10582.986291] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10589.952690] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10597.557878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10597.860813] Lustre: lustre-OST0000: new disk, initializing [10597.872952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10597.880695] Lustre: 344049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10597.977657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10603.602988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10603.613632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10603.787814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10605.437405] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10615.562969] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10621.564141] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10621.917325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10628.048879] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10628.297329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10634.724172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.726584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10634.731530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10634.743283] Lustre: Skipped 1 previous similar message [10637.011704] Lustre: server umount lustre-OST0000 complete [10643.955598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.965956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.982378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10643.985496] Lustre: Skipped 1 previous similar message [10650.080253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10650.084388] Lustre: Skipped 3 previous similar messages [10654.190079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.199206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10655.596799] Lustre: server umount lustre-MDT0000 complete [10660.204379] LustreError: 342070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851126 with bad export cookie 14465284377962918522 [10660.207399] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10660.208808] LustreError: 342070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10660.837653] Lustre: server umount lustre-MDT0001 complete [10671.542755] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:18:54 (1787851134) [10681.123385] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:19:04 (1787851144) [10689.928627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10690.392181] LustreError: 346805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10690.414360] LustreError: 346805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10690.524085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10695.650841] LustreError: 346806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.365355] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10700.773499] LustreError: 346805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10705.894309] LustreError: 346806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.727803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.659171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10713.427540] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10721.119456] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10727.569762] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10734.131626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10734.520317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10741.468574] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10748.945504] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10763.170247] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10765.319704] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10767.417401] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10769.126602] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10771.022929] Lustre: Failing over lustre-MDT0000 [10771.363524] LustreError: 348934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10773.204970] Lustre: server umount lustre-MDT0000 complete [10773.474614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10773.514292] Lustre: Skipped 3 previous similar messages [10781.557673] LustreError: 348934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.586431] LustreError: 348934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10789.855103] Lustre: 340301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851239/real 1787851239] req@000000008547ca4f x1874697622991360/t0(0) o400->MGC192.168.201.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787851255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10789.885401] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10791.688362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.935723] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.957262] LustreError: 346800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10799.462769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10799.530845] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10803.066782] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10804.740430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10804.799308] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10804.876843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10804.882161] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10815.938045] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10818.062891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10822.454244] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10824.649073] Lustre: Failing over lustre-MDT0000 [10824.950668] Lustre: server umount lustre-MDT0000 complete [10825.191264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10825.202884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10830.305331] LustreError: 346805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10830.337797] LustreError: 346805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10841.056314] Lustre: 340299:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851291/real 1787851291] req@0000000081cbdb77 x1874697623012608/t0(0) o400->MGC192.168.201.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787851307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10841.108700] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.896712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.308229] LustreError: 340298:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000006684852 x1874697623017856/t0(0) o250->MGC192.168.201.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10852.232942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10852.388434] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10855.305730] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10857.469852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10857.481794] Lustre: Skipped 2 previous similar messages [10857.526188] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10857.592673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10858.809168] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10872.204916] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10874.111687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10883.042112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10883.049608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.072765] Lustre: Skipped 4 previous similar messages [10883.086045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.103840] Lustre: Skipped 3 previous similar messages [10886.133335] Lustre: server umount lustre-MDT0000 complete [10889.660180] LustreError: 346784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851355 with bad export cookie 14465284377962921231 [10889.662740] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10889.668686] LustreError: 346784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10889.923180] Lustre: server umount lustre-MDT0001 complete [10903.727240] Lustre: server umount lustre-OST0000 complete [10921.769632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.341334] LustreError: 353177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.359325] LustreError: 353177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [10922.434511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10927.650962] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10936.464576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.060785] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10949.634934] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10955.700931] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10963.711875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10967.281877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10970.695943] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [10978.947375] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10992.101951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10992.104090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.121166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10992.164095] Lustre: Skipped 1 previous similar message [10993.282990] Lustre: server umount lustre-OST0000 complete [11004.250510] Lustre: server umount lustre-MDT0000 complete [11008.842414] LustreError: 353157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851474 with bad export cookie 14465284377962921973 [11008.851136] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11008.852037] LustreError: 353157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11009.602055] Lustre: server umount lustre-MDT0001 complete [11023.565564] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11026.487439] Key type lgssc unregistered [11026.837283] LNet: 356709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11026.848179] LNetError: 356709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11026.881996] LNet: Removed LNI 192.168.201.116@tcp [11027.913159] Key type .llcrypt unregistered [11027.915805] Key type ._llcrypt unregistered [11062.163859] Key type ._llcrypt registered [11062.166919] Key type .llcrypt registered [11062.327647] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11064.305844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11064.358042] alg: No test for adler32 (adler32-zlib) [11065.890929] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [11066.571518] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11068.391274] Key type lgssc registered [11071.128801] Lustre: Echo OBD driver; http://www.lustre.org/ [11080.818046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11080.843769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.396969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11087.181385] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11092.961572] LustreError: 358224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.981841] LustreError: 358224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11097.216851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.972818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11103.989745] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11112.603251] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11119.288771] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11126.845459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11127.567813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11128.634472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11135.096814] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11143.962723] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11158.524767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11158.535105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11158.561775] Lustre: Skipped 1 previous similar message [11158.588911] Lustre: Skipped 2 previous similar messages [11163.627584] LustreError: 358223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.662299] LustreError: 358223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11164.053394] Lustre: server umount lustre-MDT0000 complete [11168.558873] LustreError: 358204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851634 with bad export cookie 5599185687656555644 [11168.560454] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.566984] LustreError: 358204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11168.738336] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.741583] LustreError: 359488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11168.760992] Lustre: Skipped 1 previous similar message [11168.771267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11169.123566] Lustre: server umount lustre-MDT0001 complete [11176.111608] Lustre: server umount lustre-OST0000 complete [11189.232656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11189.814432] LustreError: 361959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.908511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11194.543590] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11195.366771] LustreError: 361960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.067121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.326650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11207.803185] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11216.422263] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.815434] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11233.050037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11233.912690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.965082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11240.791760] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11251.897522] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11264.994281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11265.002690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.036127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.089162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.091369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.053594] Lustre: server umount lustre-MDT0000 complete [11274.213804] LustreError: 362690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11274.256605] LustreError: 362690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11275.513781] LustreError: 361939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851741 with bad export cookie 5599185687656556757 [11275.527242] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.536740] LustreError: 361939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11276.799551] Lustre: server umount lustre-MDT0001 complete [11292.413575] Lustre: server umount lustre-OST0000 complete [11313.367707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.098667] LustreError: 365875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.294125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11319.136749] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11328.825416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.769487] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11346.799212] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11355.448396] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11365.594288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11366.234664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11366.242077] Lustre: Skipped 1 previous similar message [11374.684968] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11383.896216] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11384.647063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11400.161869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11400.162398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11400.169435] LustreError: Skipped 1 previous similar message [11400.170199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11400.170203] Lustre: Skipped 2 previous similar messages [11400.225519] Lustre: Skipped 2 previous similar messages [11401.627937] Lustre: server umount lustre-OST0000 complete [11410.400873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.418041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11410.424310] Lustre: Skipped 1 previous similar message [11412.289822] Lustre: server umount lustre-MDT0000 complete [11415.519870] LustreError: 365876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.549805] LustreError: 365876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11416.262435] LustreError: 365855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851882 with bad export cookie 5599185687656557779 [11416.275990] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11416.507852] Lustre: server umount lustre-MDT0001 complete [11428.769128] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11431.291036] Key type lgssc unregistered [11431.575296] LNet: 369409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11431.590305] LNetError: 369409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11431.601782] LNet: Removed LNI 192.168.201.116@tcp [11432.530121] Key type .llcrypt unregistered [11432.535270] Key type ._llcrypt unregistered [11459.612750] Key type ._llcrypt registered [11459.619207] Key type .llcrypt registered [11459.755301] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11460.774619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11460.902206] alg: No test for adler32 (adler32-zlib) [11461.940292] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [11462.156274] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11463.818805] Key type lgssc registered [11464.930815] Lustre: Echo OBD driver; http://www.lustre.org/ [11471.370570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11471.382781] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.388742] LustreError: 370849:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11471.409624] LustreError: 370849:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11471.415351] LustreError: 370849:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11471.420380] LustreError: 370849:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11471.427369] LustreError: 370849:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11478.498937] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.503861] LustreError: 371245:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11478.514646] LustreError: 371245:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11478.519766] LustreError: 371245:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11478.523094] LustreError: 371245:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11478.525927] LustreError: 371245:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11496.860756] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11499.877477] Key type lgssc unregistered [11500.169435] LNet: 372249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11500.174701] LNetError: 372249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11500.197929] LNet: Removed LNI 192.168.201.116@tcp [11501.015514] Key type .llcrypt unregistered [11501.017359] Key type ._llcrypt unregistered [11522.716865] Key type ._llcrypt registered [11522.720645] Key type .llcrypt registered [11522.992912] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [11546.333443] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11548.250835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11548.285745] alg: No test for adler32 (adler32-zlib) [11549.375459] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [11549.694291] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11551.447163] Key type lgssc registered [11553.219700] Lustre: Echo OBD driver; http://www.lustre.org/ [11591.326429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11591.360674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11592.700948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11592.754562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11592.870826] Lustre: lustre-MDT0000: new disk, initializing [11592.986657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11593.022087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11597.872031] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11608.455642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.593739] Lustre: 375401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11608.650239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11608.657973] Lustre: Skipped 1 previous similar message [11608.732821] Lustre: lustre-MDT0001: new disk, initializing [11608.854312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11608.905041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11608.913579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11613.339329] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11623.017979] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11629.240349] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11636.815038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.092204] Lustre: lustre-OST0000: new disk, initializing [11637.095786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11637.103067] Lustre: 377370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11637.176671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11643.813177] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11646.605525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11646.628100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11646.829965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11655.222711] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11662.222264] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11662.475724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11668.041398] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11668.226693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11672.551229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.556334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11672.572387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11672.583256] Lustre: Skipped 1 previous similar message [11672.635024] LustreError: Skipped 1 previous similar message [11676.964368] Lustre: server umount lustre-OST0000 complete [11682.790715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11682.815058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.820884] Lustre: Skipped 2 previous similar messages [11689.899235] Lustre: server umount lustre-MDT0000 complete [11693.123366] LustreError: 376198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852159 with bad export cookie 4921156909963004897 [11693.134146] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11693.507723] Lustre: server umount lustre-MDT0001 complete [11701.822568] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11703.805505] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:36:07 (1787852167) [11710.355820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.706881] LustreError: 379830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11710.745394] LustreError: 379830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11710.802899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11716.072931] LustreError: 379831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11716.834032] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11720.166756] LustreError: 379830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.289739] LustreError: 379831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.979537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.707604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.795405] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11743.597824] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11750.169616] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11759.026992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11759.452494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11769.886308] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11780.360397] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11790.679183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11790.867327] Lustre: lustre-OST0001: new disk, initializing [11790.870283] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11790.873303] Lustre: 382671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11790.940385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11798.740251] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11799.677984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11799.690829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11799.839248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11808.863453] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11815.395564] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11815.396612] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.407422] LustreError: Skipped 1 previous similar message [11815.411109] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11815.419685] Lustre: Skipped 2 previous similar messages [11818.676871] Lustre: server umount lustre-OST0001 complete [11820.521140] LustreError: 383089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.545768] LustreError: 383089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11830.774041] LustreError: 383089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11830.824732] LustreError: 383089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11840.480885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11840.484972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.495915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11840.498991] Lustre: Skipped 2 previous similar messages [11843.670995] Lustre: server umount lustre-OST0000 complete [11849.184616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11849.194319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11849.207258] Lustre: Skipped 2 previous similar messages [11849.236605] Lustre: Skipped 2 previous similar messages [11856.667355] Lustre: server umount lustre-MDT0000 complete [11859.429502] LustreError: 379830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11859.439028] LustreError: 379830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11860.740477] LustreError: 379812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852326 with bad export cookie 4921156909963005779 [11860.747518] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11860.763297] LustreError: 379812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11875.295176] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11875.695237] Lustre: server umount lustre-MDT0001 complete [11888.346842] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [11893.228704] Key type lgssc unregistered [11893.751736] LNet: 384663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11893.760730] LNetError: 384663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11893.790229] LNet: Removed LNI 192.168.201.116@tcp [11895.116964] Key type .llcrypt unregistered [11895.120301] Key type ._llcrypt unregistered [11957.165398] Key type ._llcrypt registered [11957.167959] Key type .llcrypt registered [11957.283574] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [11958.680255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11958.702534] alg: No test for adler32 (adler32-zlib) [11959.809402] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [11960.090210] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [11961.815166] Key type lgssc registered [11963.017385] Lustre: Echo OBD driver; http://www.lustre.org/ [11969.814571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11969.851297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.140247] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11971.166826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11971.302264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11975.713749] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11983.745775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11983.818381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11983.847734] Lustre: 387466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11984.116202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11988.917194] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [11997.127836] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12003.590956] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12009.456444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.460075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.489957] Lustre: Skipped 1 previous similar message [12009.514988] Lustre: Skipped 1 previous similar message [12012.019220] Lustre: server umount lustre-MDT0000 complete [12016.489337] LustreError: 387458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852482 with bad export cookie 361423865227999633 [12016.501056] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12017.441584] Lustre: server umount lustre-MDT0001 complete [12027.369721] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:41:31 (1787852491) [12033.809017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12034.363367] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12034.373195] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12034.420277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12038.447308] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12039.656962] LustreError: 389978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.770575] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.067975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.448594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12052.896625] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12060.410398] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12066.248630] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12072.699917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12073.025222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12073.285247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12082.992403] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12094.170730] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12104.628983] Lustre: Permanently deactivating lustre-OST0000 [12104.646692] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12113.951356] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12116.400927] LustreError: 393015:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12116.404869] LustreError: 393015:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12116.408319] LustreError: 393015:0:(mgs_handler.c:1286:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12119.248947] LustreError: 393064:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12119.258172] LustreError: 393064:0:(mgs_handler.c:1286:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12127.425943] Lustre: server umount lustre-OST0000 complete [12130.823434] LustreError: 393314:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12130.840893] LustreError: 393314:0:(mgs_handler.c:1286:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12134.368715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12134.377252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.396858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.394178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.397466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.420730] Lustre: Skipped 1 previous similar message [12140.524867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12140.535353] Lustre: Skipped 1 previous similar message [12142.750510] Lustre: server umount lustre-MDT0000 complete [12145.652876] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12145.693881] LustreError: 389977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12149.174906] LustreError: 389960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852615 with bad export cookie 361423865228000214 [12149.178201] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12149.185797] LustreError: 389960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12159.665855] Lustre: server umount lustre-MDT0001 complete [12168.683601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12173.166363] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12175.767611] LustreError: 394476:0:(mgs_handler.c:1286:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12181.603509] Lustre: 394815:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.116@tcp [12183.496647] Lustre: server umount MGS complete [12193.973893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12194.670265] LustreError: 395538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12194.697238] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12194.702588] Lustre: Skipped 1 previous similar message [12194.775529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12201.723937] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12205.024204] LustreError: 395539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12205.057288] LustreError: 395539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12216.580385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12217.398134] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12217.571134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12223.473298] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12232.671835] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12240.090440] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12241.834365] Lustre: Permanently reactivating lustre-OST0000 [12241.856301] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12241.862448] Lustre: Skipped 2 previous similar messages [12269.536716] LustreError: 397271:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12277.482531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.942368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12284.707862] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12291.111796] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12299.749224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12299.756096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12299.756467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12299.758971] LustreError: Skipped 1 previous similar message [12299.765577] Lustre: Skipped 1 previous similar message [12301.606328] Lustre: server umount lustre-OST0000 complete [12309.476973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12309.501917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12309.517085] Lustre: Skipped 1 previous similar message [12309.525991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12311.737168] Lustre: server umount lustre-MDT0000 complete [12315.107465] LustreError: 395539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12315.125857] LustreError: 395539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12315.373802] LustreError: 398852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852781 with bad export cookie 361423865228001264 [12315.376853] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12315.381815] LustreError: 398852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12315.970433] Lustre: server umount lustre-MDT0001 complete [12331.248686] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12334.979931] Key type lgssc unregistered [12335.416064] LNet: 399808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12335.418953] LNetError: 399808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12335.447711] LNet: Removed LNI 192.168.201.116@tcp [12336.428997] Key type .llcrypt unregistered [12336.431188] Key type ._llcrypt unregistered [12354.608472] Key type ._llcrypt registered [12354.611279] Key type .llcrypt registered [12354.811566] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [12371.197705] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [12372.216182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12372.295155] alg: No test for adler32 (adler32-zlib) [12373.257353] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [12373.427111] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [12375.057527] Key type lgssc registered [12376.913890] Lustre: Echo OBD driver; http://www.lustre.org/ [12414.022378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12414.082764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12415.439042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12415.514465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12415.602687] Lustre: lustre-MDT0000: new disk, initializing [12415.685632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12415.695793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12420.437671] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12432.734730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.014058] Lustre: 402962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12433.184205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12433.190821] Lustre: Skipped 1 previous similar message [12433.394307] Lustre: lustre-MDT0001: new disk, initializing [12433.566662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12433.610169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12433.629861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12439.684325] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12451.468746] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12459.578935] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12467.177270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12467.449375] Lustre: lustre-OST0000: new disk, initializing [12467.452781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12467.459927] Lustre: 404929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12467.517567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12474.227487] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12474.978017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12474.993485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12475.124228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12484.903709] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12490.326910] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12490.569349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12495.071154] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12495.307469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12500.962817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12500.980505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12500.991046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12501.013793] Lustre: Skipped 1 previous similar message [12503.814710] Lustre: server umount lustre-OST0000 complete [12510.192456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12510.204832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12510.215713] Lustre: Skipped 1 previous similar message [12510.222984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12513.848615] Lustre: server umount lustre-MDT0000 complete [12517.382907] LustreError: 402954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852983 with bad export cookie 5037014902723951760 [12517.392517] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12517.650921] Lustre: server umount lustre-MDT0001 complete [12525.685887] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12527.568591] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:49:51 (1787852991) [12534.958263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12535.328427] LustreError: 407391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.345851] LustreError: 407391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12535.399443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12540.318657] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12540.397091] LustreError: 407392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.509538] LustreError: 407391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.707884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.090709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12553.285827] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12560.829913] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12566.231847] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12573.216824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12573.514496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12580.490183] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12588.076434] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12590.278667] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12592.991557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12600.295265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12600.303162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12600.316939] Lustre: Skipped 1 previous similar message [12600.321907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12600.327939] Lustre: Skipped 2 previous similar messages [12604.406516] Lustre: server umount lustre-OST0000 complete [12610.528274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12610.530752] LustreError: Skipped 1 previous similar message [12610.541245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.556539] Lustre: Skipped 1 previous similar message [12610.560442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12610.573985] Lustre: Skipped 2 previous similar messages [12615.885255] Lustre: server umount lustre-MDT0000 complete [12616.676164] LustreError: 407391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.702312] LustreError: 407391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12620.038487] LustreError: 407372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853086 with bad export cookie 5037014902723952642 [12620.045986] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12620.058954] LustreError: 407372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12620.384337] Lustre: server umount lustre-MDT0001 complete [12633.651385] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12637.305753] Key type lgssc unregistered [12637.830661] LNet: 411114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12637.844911] LNetError: 411114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12637.863856] LNet: Removed LNI 192.168.201.116@tcp [12639.209841] Key type .llcrypt unregistered [12639.218425] Key type ._llcrypt unregistered [12663.993152] Key type ._llcrypt registered [12663.997982] Key type .llcrypt registered [12664.098841] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12664.408215] Key type .llcrypt unregistered [12664.412369] Key type ._llcrypt unregistered [12685.657420] Key type ._llcrypt registered [12685.659411] Key type .llcrypt registered [12685.911505] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [12687.646640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12687.810805] alg: No test for adler32 (adler32-zlib) [12689.083751] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [12689.473404] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [12691.201789] Key type lgssc registered [12692.514328] Lustre: Echo OBD driver; http://www.lustre.org/ [12701.859334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12701.887920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12703.323680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12709.268957] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12713.444053] LustreError: 413323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.455491] LustreError: 413323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12718.575426] LustreError: 413324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.512938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12720.812814] LustreError: 413323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.865042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12726.248381] LustreError: 413319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12726.268882] LustreError: 413319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12728.305829] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12728.326803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12733.432164] LustreError: 413320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12733.462173] LustreError: 413320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12737.456690] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12743.650498] LustreError: 413323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.682796] LustreError: 413323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12745.046741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12745.516513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12749.938186] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12759.017697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12759.028781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12759.047959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12760.951371] Lustre: server umount lustre-OST0000 complete [12766.178310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12766.201367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12766.202606] Lustre: Skipped 1 previous similar message [12766.216640] Lustre: Skipped 1 previous similar message [12767.367447] Lustre: server umount lustre-MDT0000 complete [12771.298082] LustreError: 413324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12771.315558] LustreError: 413324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12771.618828] LustreError: 415068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853237 with bad export cookie 17689361077829447185 [12771.631809] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12771.632303] LustreError: 415068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12781.312965] Lustre: server umount lustre-MDT0001 complete [12795.555636] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12799.247843] Key type lgssc unregistered [12799.565879] LNet: 416472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12799.571605] LNetError: 416472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12800.622761] LNet: Removed LNI 192.168.201.116@tcp [12801.761309] Key type .llcrypt unregistered [12801.769729] Key type ._llcrypt unregistered [12826.497765] Key type ._llcrypt registered [12826.503869] Key type .llcrypt registered [12826.633631] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [12828.193395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12828.230972] alg: No test for adler32 (adler32-zlib) [12829.292993] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [12829.641363] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [12831.392249] Key type lgssc registered [12832.866334] Lustre: Echo OBD driver; http://www.lustre.org/ [12840.934312] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12840.964872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12856.607589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12861.727374] LustreError: 417859:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [12888.480412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12896.041444] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12905.101926] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12912.726515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12913.176647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12913.229034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12918.253582] LustreError: 417884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12918.285207] LustreError: 417884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12919.430208] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12923.364894] LustreError: 418894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12923.384625] LustreError: 418894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12928.488686] LustreError: 418897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12928.528635] LustreError: 418897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12928.819672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12929.201667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12933.811379] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [12941.200538] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12947.066671] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12955.107796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.107919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12955.123620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12956.674844] Lustre: server umount lustre-OST0000 complete [12961.766753] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12961.774803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12961.782412] Lustre: Skipped 1 previous similar message [12961.798817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12961.805605] Lustre: Skipped 1 previous similar message [12963.026646] Lustre: server umount lustre-MDT0000 complete [12965.344775] LustreError: 420659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12965.350972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.825252] LustreError: 417866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853432 with bad export cookie 3813325568180617196 [12966.837831] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12966.848616] LustreError: 417866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12967.266107] Lustre: server umount lustre-MDT0001 complete [12978.394381] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [12981.739601] Key type lgssc unregistered [12982.058792] LNet: 421428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12982.062506] LNetError: 421428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12983.077908] LNet: Removed LNI 192.168.201.116@tcp [12984.187984] Key type .llcrypt unregistered [12984.190147] Key type ._llcrypt unregistered [13008.206761] Key type ._llcrypt registered [13008.211488] Key type .llcrypt registered [13008.367570] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13009.603644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13009.630558] alg: No test for adler32 (adler32-zlib) [13010.795745] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [13011.093846] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [13012.802867] Key type lgssc registered [13014.460798] Lustre: Echo OBD driver; http://www.lustre.org/ [13022.701492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13022.753401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13024.362712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13029.348990] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13034.976530] LustreError: 422847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.998123] LustreError: 422847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13038.935509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13039.500126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13044.466583] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13053.109461] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13058.096258] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13065.944766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13066.287774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13070.386284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13072.656300] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13081.196183] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13089.186643] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [13089.761583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13089.763124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13089.777241] Lustre: Skipped 2 previous similar messages [13089.783451] Lustre: Skipped 1 previous similar message [13092.904213] Lustre: server umount lustre-MDT0000 complete [13094.253166] LustreError: 425294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13095.320163] LustreError: 422829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.16@tcp arrived at 1787853561 with bad export cookie 7127875548681370846 [13095.335128] LustreError: 422829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13098.056048] LustreError: 425426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853564 with bad export cookie 7127875548681369957 [13098.061054] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13098.074451] LustreError: 425426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13100.003056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.004592] LustreError: 422843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13100.060687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13100.081877] LustreError: 422843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13103.014379] LustreError: 422847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13103.017193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13103.057130] LustreError: 422847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13104.278250] Lustre: server umount lustre-OST0000 complete [13108.133290] LustreError: 422848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13108.157449] LustreError: 422848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13111.744503] Lustre: server umount lustre-MDT0001 complete [13124.649133] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13127.180453] Key type lgssc unregistered [13127.524653] LNet: 426531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13127.537816] LNetError: 426531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13127.576163] LNet: Removed LNI 192.168.201.116@tcp [13128.582972] Key type .llcrypt unregistered [13128.587156] Key type ._llcrypt unregistered [13153.299555] Key type ._llcrypt registered [13153.301698] Key type .llcrypt registered [13153.407299] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13154.485193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13154.558483] alg: No test for adler32 (adler32-zlib) [13155.577576] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [13155.795895] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [13157.431201] Key type lgssc registered [13158.586139] Lustre: Echo OBD driver; http://www.lustre.org/ [13166.558982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13166.601345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13168.100335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13172.827586] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13178.343973] LustreError: 427942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.376972] LustreError: 427942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13182.290576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13182.780710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13187.930511] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13195.506201] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13202.156226] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13209.808944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13210.136556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13213.248818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13217.050578] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13224.973262] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13232.622726] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13232.639830] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13232.645715] Lustre: Skipped 1 previous similar message [13236.124415] Lustre: server umount lustre-MDT0001 complete [13237.733182] LustreError: 427943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13237.768486] LustreError: 427943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13242.852670] LustreError: 427943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13242.854155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13242.884039] LustreError: 427943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13242.903320] Lustre: Skipped 1 previous similar message [13242.916115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13242.931613] Lustre: Skipped 1 previous similar message [13247.973917] LustreError: 427942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13247.986073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13249.348083] Lustre: server umount lustre-OST0000 complete [13253.091706] LustreError: 427943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13254.302410] Lustre: server umount lustre-MDT0000 complete [13268.770248] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13271.480696] Key type lgssc unregistered [13271.806093] LNet: 431634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13271.828227] LNetError: 431634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13271.850449] LNet: Removed LNI 192.168.201.116@tcp [13272.831542] Key type .llcrypt unregistered [13272.833658] Key type ._llcrypt unregistered [13301.917606] Key type ._llcrypt registered [13301.919566] Key type .llcrypt registered [13302.013651] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [13317.118116] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13318.705784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13318.763104] alg: No test for adler32 (adler32-zlib) [13319.822415] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [13320.076865] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [13321.745454] Key type lgssc registered [13322.965687] Lustre: Echo OBD driver; http://www.lustre.org/ [13370.497454] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13387.787637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13387.812302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13389.148650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13389.191842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13389.329595] Lustre: lustre-MDT0000: new disk, initializing [13389.468063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13389.493640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13396.018451] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13413.934351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13414.355884] Lustre: lustre-OST0000: new disk, initializing [13414.358834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13414.364035] Lustre: Skipped 1 previous similar message [13414.370157] Lustre: 436668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13414.461706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13420.192873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13420.202915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13420.259921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13421.716896] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13431.963259] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13445.551214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13445.795331] Lustre: 435649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13445.997677] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (not set up) [13445.998365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13446.121808] Lustre: lustre-MDT0001: new disk, initializing [13446.255362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13446.300546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13446.319463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13451.675658] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13461.880412] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13462.095344] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13466.216602] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13466.396466] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13471.629949] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [13474.275579] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13474.286750] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13474.299116] Lustre: Skipped 1 previous similar message [13474.300072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13476.722629] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [13476.729516] Lustre: Skipped 1 previous similar message [13478.077178] Lustre: server umount lustre-MDT0001 complete [13484.528931] LustreError: 435657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13484.549679] LustreError: 435657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13489.655836] LustreError: 435660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13489.674530] LustreError: 435660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13494.782448] LustreError: 435655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13494.799813] LustreError: 435655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13497.312708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13497.318719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13497.341278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13499.879299] LustreError: 435660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.817338] Lustre: server umount lustre-OST0000 complete [13505.016636] LustreError: 438364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.801535] Lustre: server umount lustre-MDT0000 complete [13528.437227] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13532.444771] Key type lgssc unregistered [13532.891720] LNet: 440419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13532.904211] LNetError: 440419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13532.918548] LNet: Removed LNI 192.168.201.116@tcp [13533.883539] Key type .llcrypt unregistered [13533.886186] Key type ._llcrypt unregistered [13576.025772] Key type ._llcrypt registered [13576.027250] Key type .llcrypt registered [13576.119727] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [13576.449390] Key type .llcrypt unregistered [13576.451899] Key type ._llcrypt unregistered [13591.386450] Key type ._llcrypt registered [13591.388707] Key type .llcrypt registered [13591.473181] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [13592.277812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13592.300041] alg: No test for adler32 (adler32-zlib) [13593.521192] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [13593.726909] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [13595.399398] Key type lgssc registered [13596.474433] Lustre: Echo OBD driver; http://www.lustre.org/ [13621.078694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13621.130805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13622.425241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13622.452325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13622.548402] Lustre: lustre-MDT0000: new disk, initializing [13622.608870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13622.626038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13627.560496] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13639.887108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13640.014183] Lustre: 443785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13640.079743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13640.086254] Lustre: Skipped 1 previous similar message [13640.193751] Lustre: lustre-MDT0001: new disk, initializing [13640.314402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13640.375328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13640.398300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13645.066304] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13654.556778] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13661.365262] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13669.110812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13669.401910] Lustre: lustre-OST0000: new disk, initializing [13669.406753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13669.411683] Lustre: 445753:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13669.489198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13676.120774] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13677.178209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13677.187436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13677.423613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13685.321530] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13692.897228] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13692.912627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13692.914665] Lustre: Skipped 2 previous similar messages [13692.931878] Lustre: Skipped 2 previous similar messages [13698.017090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13698.030311] Lustre: Skipped 2 previous similar messages [13698.669450] Lustre: server umount lustre-MDT0000 complete [13702.801112] LustreError: 443777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854168 with bad export cookie 12688999435931120264 [13702.815220] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13703.122281] Lustre: server umount lustre-MDT0001 complete [13717.374281] Lustre: server umount lustre-OST0000 complete [13732.703212] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:09:55 (1787854195) [13742.213624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13742.703992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13746.963384] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13748.204446] LustreError: 447786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13748.227172] LustreError: 447786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13752.291256] LustreError: 447785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13755.797200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13756.094393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13760.241025] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13766.769490] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13772.290352] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13793.263375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13793.504641] Lustre: Found index 0 for lustre-OST0000, updating log [13793.703856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13801.170948] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [13808.062803] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13813.188822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13823.456326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13823.466618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13823.473255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13823.491869] Lustre: Skipped 1 previous similar message [13827.643978] Lustre: server umount lustre-OST0000 complete [13845.094795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13845.119350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13845.127527] Lustre: Skipped 4 previous similar messages [13850.276217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13850.302163] Lustre: Skipped 1 previous similar message [13851.477239] Lustre: server umount lustre-MDT0000 complete [13855.424940] LustreError: 447765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854321 with bad export cookie 12688999435931121657 [13855.436304] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13855.440119] LustreError: 447765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13855.875108] Lustre: server umount lustre-MDT0001 complete [13856.991266] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [13865.873422] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:12:09 (1787854329) [14486.519235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14486.968787] LustreError: 454216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14487.022429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14490.987235] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [14492.128811] LustreError: 454217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14492.149447] LustreError: 454217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14496.229364] LustreError: 454216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14499.665732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14499.971350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14504.836081] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [14511.485908] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14517.159749] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14537.662101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14537.951234] Lustre: Found index 0 for lustre-OST0000, updating log [14538.223236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14544.224150] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [14548.255299] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [14550.717141] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14552.553837] Lustre: server umount lustre-OST0000 complete [14556.091947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14556.109799] Lustre: Skipped 1 previous similar message [14556.120847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14556.129696] Lustre: Skipped 2 previous similar messages [14557.153343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14557.162547] Lustre: Skipped 2 previous similar messages [14559.200781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14559.204502] Lustre: Skipped 3 previous similar messages [14562.393470] Lustre: server umount lustre-MDT0000 complete [14563.297175] LustreError: 455225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14563.315082] LustreError: 455225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14565.621919] LustreError: 454197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855031 with bad export cookie 12688999435931122742 [14565.624248] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14565.635413] LustreError: 454197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14565.984975] Lustre: server umount lustre-MDT0001 complete [14595.028381] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [14604.118423] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [14638.070300] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:25:02 (1787855102) [14656.213729] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [14658.667606] Key type lgssc unregistered [14658.915041] LNet: 460829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14658.920310] LNetError: 460829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14746.984387] LNet: Removed LNI 192.168.201.116@tcp [14748.266981] Key type .llcrypt unregistered [14748.270338] Key type ._llcrypt unregistered [14765.904447] Key type ._llcrypt registered [14765.909646] Key type .llcrypt registered [14766.009260] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [14767.521529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14767.574233] alg: No test for adler32 (adler32-zlib) [14768.662424] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [14768.859842] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [14770.536666] Key type lgssc registered [14772.182717] Lustre: Echo OBD driver; http://www.lustre.org/ [14776.759787] LNet: Added LNI 192.168.201.116@tcp100 [8/256/0/180] [14777.669743] LNet: Added LNI 192.168.201.116@tcp101 [8/256/0/180] [14778.742728] LNet: Added LNI 192.168.201.116@tcp102 [8/256/0/180] [14779.740285] LNet: Added LNI 192.168.201.116@tcp103 [8/256/0/180] [14780.539875] LNet: Added LNI 192.168.201.116@tcp104 [8/256/0/180] [14781.281440] LNet: Added LNI 192.168.201.116@tcp105 [8/256/0/180] [14782.230296] LNet: Added LNI 192.168.201.116@tcp106 [8/256/0/180] [14782.947166] LNet: Added LNI 192.168.201.116@tcp107 [8/256/0/180] [14783.857611] LNet: Added LNI 192.168.201.116@tcp108 [8/256/0/180] [14784.926147] LNet: Added LNI 192.168.201.116@tcp109 [8/256/0/180] [14786.208354] LNet: Added LNI 192.168.201.116@tcp110 [8/256/0/180] [14787.111855] LNet: Added LNI 192.168.201.116@tcp111 [8/256/0/180] [14788.168140] LNet: Added LNI 192.168.201.116@tcp112 [8/256/0/180] [14789.464776] LNet: Added LNI 192.168.201.116@tcp113 [8/256/0/180] [14790.497930] LNet: Added LNI 192.168.201.116@tcp114 [8/256/0/180] [14791.616495] LNet: Added LNI 192.168.201.116@tcp115 [8/256/0/180] [14792.790740] LNet: Added LNI 192.168.201.116@tcp116 [8/256/0/180] [14793.918311] LNet: Added LNI 192.168.201.116@tcp117 [8/256/0/180] [14795.250665] LNet: Added LNI 192.168.201.116@tcp118 [8/256/0/180] [14796.959384] LNet: Added LNI 192.168.201.116@tcp119 [8/256/0/180] [14798.409643] LNet: Added LNI 192.168.201.116@tcp120 [8/256/0/180] [14799.879767] LNet: Added LNI 192.168.201.116@tcp121 [8/256/0/180] [14800.908204] LNet: Added LNI 192.168.201.116@tcp122 [8/256/0/180] [14802.281281] LNet: Added LNI 192.168.201.116@tcp123 [8/256/0/180] [14803.252637] LNet: Added LNI 192.168.201.116@tcp124 [8/256/0/180] [14804.386953] LNet: Added LNI 192.168.201.116@tcp125 [8/256/0/180] [14806.268701] LNet: Added LNI 192.168.201.116@tcp126 [8/256/0/180] [14807.801575] LNet: Added LNI 192.168.201.116@tcp127 [8/256/0/180] [14809.048418] LNet: Added LNI 192.168.201.116@tcp128 [8/256/0/180] [14810.247778] LNet: Added LNI 192.168.201.116@tcp129 [8/256/0/180] [14811.240660] LNet: Added LNI 192.168.201.116@tcp130 [8/256/0/180] [14811.987436] LNet: Added LNI 192.168.201.116@tcp131 [8/256/0/180] [14813.001601] LNet: Added LNI 192.168.201.116@tcp132 [8/256/0/180] [14814.578301] LNet: Added LNI 192.168.201.116@tcp133 [8/256/0/180] [14815.260839] LNet: Added LNI 192.168.201.116@tcp134 [8/256/0/180] [14816.045651] LNet: Added LNI 192.168.201.116@tcp135 [8/256/0/180] [14822.457774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14822.485914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14823.635722] Lustre: 463856:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [14823.718759] LNetError: 463879:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14823.750933] LNetError: Protocol error connecting to 192.168.201.116@tcp118 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14823.771693] LNetError: Skipped 5 previous similar messages [14823.892462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14823.920616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14823.984902] Lustre: lustre-MDT0000: new disk, initializing [14824.051516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14824.065828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14824.226892] LNet: 6 local NIs in recovery (showing 6): 192.168.201.116@tcp125, 192.168.201.116@tcp126, 192.168.201.116@tcp127, 192.168.201.116@tcp118, 192.168.201.116@tcp128, 192.168.201.116@tcp129 [14824.237519] LNetError: 463879:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14824.273136] LNetError: 463879:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14824.283166] LNetError: Protocol error connecting to 192.168.201.116@tcp125 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14824.291567] LNetError: Skipped 6 previous similar messages [14825.313899] LNetError: 461312:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14825.329391] LNetError: 461312:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14825.343394] LNetError: Protocol error connecting to 192.168.201.116@tcp128 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14825.371734] LNetError: Skipped 1 previous similar message [14827.373643] LNetError: 463876:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14827.388944] LNetError: 463876:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14827.402553] LNetError: Protocol error connecting to 192.168.201.116@tcp128 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14827.413062] LNetError: Skipped 1 previous similar message [14829.378364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [14831.458939] LNetError: 463877:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14831.477979] LNetError: 463877:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14831.484371] LNetError: Protocol error connecting to 192.168.201.116@tcp128 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14831.501713] LNetError: Skipped 1 previous similar message [14839.651066] LNetError: 463889:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14839.665583] LNetError: 463889:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14839.674616] LNetError: Protocol error connecting to 192.168.201.116@tcp128 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14839.683444] LNetError: Skipped 1 previous similar message [14841.341574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14841.502281] Lustre: 463873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14841.559454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14841.563429] Lustre: Skipped 1 previous similar message [14841.641227] Lustre: lustre-MDT0001: new disk, initializing [14841.682058] Lustre: 464675:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [14841.691558] Lustre: 464675:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14841.841993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14841.876316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14841.889580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14845.746618] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [14849.121403] Lustre: 465064:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [14849.143096] Lustre: 465064:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14855.904496] LNetError: 461313:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14855.921753] LNetError: 461313:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 22 previous similar messages [14855.929887] LNetError: Protocol error connecting to 192.168.201.116@tcp124 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14855.946061] LNetError: Skipped 22 previous similar messages [14856.242498] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14861.937133] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14868.409195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14868.796645] Lustre: lustre-OST0000: new disk, initializing [14868.801112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14868.807337] Lustre: 465848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14868.943995] Lustre: 465853:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [14868.951523] Lustre: 465853:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14869.018092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14875.873490] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [14876.216297] Lustre: 466224:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [14876.222797] Lustre: 466224:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14876.295214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14876.304688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14876.443481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14884.705282] LNet: 4 local NIs in recovery (showing 4): 192.168.201.116@tcp134, 192.168.201.116@tcp135, 192.168.201.116@tcp128, 192.168.201.116@tcp125 [14886.671524] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14888.675149] LNetError: 463879:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [14888.707205] LNetError: 463879:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 30 previous similar messages [14888.712960] LNetError: Protocol error connecting to 192.168.201.116@tcp115 at host 192.168.201.116:988: is it running a compatible version of Lustre? [14888.743364] LNetError: Skipped 30 previous similar messages [14897.122423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14897.129980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14897.137256] LustreError: Skipped 1 previous similar message [14897.141233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14897.160517] Lustre: Skipped 1 previous similar message [14897.184329] Lustre: Skipped 1 previous similar message [14901.526409] Lustre: server umount lustre-OST0000 complete [14907.362440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14907.393646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14907.397659] Lustre: Skipped 1 previous similar message [14907.414326] Lustre: Skipped 1 previous similar message [14912.415198] Lustre: server umount lustre-MDT0000 complete [14915.776506] LustreError: 463865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855381 with bad export cookie 2377020905092369535 [14915.787299] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14915.791974] LustreError: 463865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14916.137412] Lustre: server umount lustre-MDT0001 complete [14928.118256] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [14930.795857] Key type lgssc unregistered [14931.261483] LNet: 467763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14931.271661] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14932.320197] LNet: Removed LNI 192.168.201.116@tcp [14932.329769] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14932.348592] LNet: Removed LNI 192.168.201.116@tcp100 [14933.407215] LNet: Removed LNI 192.168.201.116@tcp101 [14933.420934] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14933.431608] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14933.453491] LNet: Removed LNI 192.168.201.116@tcp102 [14934.496335] LNet: Removed LNI 192.168.201.116@tcp103 [14935.583373] LNet: Removed LNI 192.168.201.116@tcp104 [14935.584826] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14935.592033] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14935.600729] LNet: Removed LNI 192.168.201.116@tcp105 [14936.607195] LNet: Removed LNI 192.168.201.116@tcp106 [14937.632361] LNet: Removed LNI 192.168.201.116@tcp107 [14938.656437] LNet: Removed LNI 192.168.201.116@tcp108 [14939.679444] LNet: Removed LNI 192.168.201.116@tcp109 [14939.708902] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14939.721681] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14939.730596] LNet: Removed LNI 192.168.201.116@tcp110 [14940.767371] LNet: Removed LNI 192.168.201.116@tcp111 [14941.791514] LNet: Removed LNI 192.168.201.116@tcp112 [14942.815214] LNet: Removed LNI 192.168.201.116@tcp113 [14943.843909] LNet: Removed LNI 192.168.201.116@tcp114 [14944.863233] LNet: Removed LNI 192.168.201.116@tcp115 [14945.889742] LNet: Removed LNI 192.168.201.116@tcp116 [14946.911482] LNet: Removed LNI 192.168.201.116@tcp117 [14947.935261] LNet: Removed LNI 192.168.201.116@tcp118 [14947.955626] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14947.966656] LNetError: 467763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14947.978832] LNet: Removed LNI 192.168.201.116@tcp119 [14949.023774] LNet: Removed LNI 192.168.201.116@tcp120 [14950.047228] LNet: Removed LNI 192.168.201.116@tcp121 [14951.071188] LNet: Removed LNI 192.168.201.116@tcp122 [14952.159453] LNet: Removed LNI 192.168.201.116@tcp123 [14953.183440] LNet: Removed LNI 192.168.201.116@tcp124 [14953.193574] LNet: Removed LNI 192.168.201.116@tcp125 [14954.271447] LNet: Removed LNI 192.168.201.116@tcp126 [14955.297867] LNet: Removed LNI 192.168.201.116@tcp127 [14955.304283] LNet: Removed LNI 192.168.201.116@tcp128 [14956.319235] LNet: Removed LNI 192.168.201.116@tcp129 [14957.343217] LNet: Removed LNI 192.168.201.116@tcp130 [14958.370076] LNet: Removed LNI 192.168.201.116@tcp131 [14959.391179] LNet: Removed LNI 192.168.201.116@tcp132 [14960.417877] LNet: Removed LNI 192.168.201.116@tcp133 [14960.422080] LNet: Removed LNI 192.168.201.116@tcp134 [14960.442249] LNet: Removed LNI 192.168.201.116@tcp135 [14961.537617] Key type .llcrypt unregistered [14961.539676] Key type ._llcrypt unregistered [14992.070231] Key type ._llcrypt registered [14992.072488] Key type .llcrypt registered [14992.180560] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [15009.098359] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [15010.000068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15010.005728] alg: No test for adler32 (adler32-zlib) [15011.048438] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [15011.518464] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [15013.298069] Key type lgssc registered [15014.727948] Lustre: Echo OBD driver; http://www.lustre.org/ [15054.988479] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:31:58 (1787855518) [15075.097931] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [15077.580926] Key type lgssc unregistered [15077.828584] LNet: 471852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15077.837755] LNetError: 471852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15077.859869] LNet: Removed LNI 192.168.201.116@tcp [15078.704660] Key type .llcrypt unregistered [15078.708879] Key type ._llcrypt unregistered [15095.526066] Key type ._llcrypt registered [15095.527239] Key type .llcrypt registered [15095.682547] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [15096.602230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15096.669073] alg: No test for adler32 (adler32-zlib) [15097.968855] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [15098.323471] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [15100.047174] Key type lgssc registered [15101.200049] Lustre: Echo OBD driver; http://www.lustre.org/ [15111.878618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15111.910216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15113.423186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15113.460918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15113.528425] Lustre: lustre-MDT0000: new disk, initializing [15113.638994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15113.673659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15118.841194] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15130.205194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15130.334406] Lustre: 473061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15130.362221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15130.364423] Lustre: Skipped 1 previous similar message [15130.425433] Lustre: lustre-MDT0001: new disk, initializing [15130.502214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15130.545854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15130.560935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15136.818812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15146.520431] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15152.903858] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15160.725206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15161.094197] Lustre: lustre-OST0000: new disk, initializing [15161.116661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15161.122802] Lustre: 475033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15161.302980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15168.615241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15168.639648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15168.841115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15168.962721] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15179.310348] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15183.374747] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing lnet_if_list [15188.802732] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15188.831837] LNet: Added LNI 192.168.201.116@tcp10 [8/256/0/180] [15194.762975] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15194.796555] LNet: Added LNI 192.168.201.116@tcp11 [8/256/0/180] [15199.006400] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15199.031703] LNet: Added LNI 192.168.201.116@tcp12 [8/256/0/180] [15203.532364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15203.589206] LNet: Added LNI 192.168.201.116@tcp13 [8/256/0/180] [15207.798290] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15207.827572] LNet: Added LNI 192.168.201.116@tcp14 [8/256/0/180] [15212.415785] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15212.446032] LNet: Added LNI 192.168.201.116@tcp15 [8/256/0/180] [15217.040953] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15217.065082] LNet: Added LNI 192.168.201.116@tcp16 [8/256/0/180] [15221.699772] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15221.761058] LNet: Added LNI 192.168.201.116@tcp17 [8/256/0/180] [15226.708344] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15226.758846] LNet: Added LNI 192.168.201.116@tcp18 [8/256/0/180] [15231.621793] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15231.655809] LNet: Added LNI 192.168.201.116@tcp19 [8/256/0/180] [15236.578302] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15236.615598] LNet: Added LNI 192.168.201.116@tcp20 [8/256/0/180] [15241.810619] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15241.831665] LNet: Added LNI 192.168.201.116@tcp21 [8/256/0/180] [15247.787874] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15247.831682] LNet: Added LNI 192.168.201.116@tcp22 [8/256/0/180] [15253.929954] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15253.952639] LNet: Added LNI 192.168.201.116@tcp23 [8/256/0/180] [15260.364797] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15260.401791] LNet: Added LNI 192.168.201.116@tcp24 [8/256/0/180] [15266.556593] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15266.588777] LNet: Added LNI 192.168.201.116@tcp25 [8/256/0/180] [15273.556369] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15273.600894] LNet: Added LNI 192.168.201.116@tcp26 [8/256/0/180] [15279.296877] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15279.319651] LNet: Added LNI 192.168.201.116@tcp27 [8/256/0/180] [15285.397773] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15285.442283] LNet: Added LNI 192.168.201.116@tcp28 [8/256/0/180] [15291.339410] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15291.377101] LNet: Added LNI 192.168.201.116@tcp29 [8/256/0/180] [15297.162553] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15297.187828] LNet: Added LNI 192.168.201.116@tcp30 [8/256/0/180] [15304.132920] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15304.162904] LNet: Added LNI 192.168.201.116@tcp31 [8/256/0/180] [15311.177338] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15311.221074] LNet: Added LNI 192.168.201.116@tcp32 [8/256/0/180] [15320.481300] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15320.513908] LNet: Added LNI 192.168.201.116@tcp33 [8/256/0/180] [15327.349749] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15327.385743] LNet: Added LNI 192.168.201.116@tcp34 [8/256/0/180] [15334.449842] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15334.475478] LNet: Added LNI 192.168.201.116@tcp35 [8/256/0/180] [15342.266379] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15342.288807] LNet: Added LNI 192.168.201.116@tcp36 [8/256/0/180] [15349.785290] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15349.835520] LNet: Added LNI 192.168.201.116@tcp37 [8/256/0/180] [15356.933632] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15356.976984] LNet: Added LNI 192.168.201.116@tcp38 [8/256/0/180] [15364.131946] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15364.164740] LNet: Added LNI 192.168.201.116@tcp39 [8/256/0/180] [15373.590463] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15373.651098] LNet: Added LNI 192.168.201.116@tcp40 [8/256/0/180] [15383.268170] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15383.329602] LNet: Added LNI 192.168.201.116@tcp41 [8/256/0/180] [15393.851514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15393.952610] LNet: Added LNI 192.168.201.116@tcp42 [8/256/0/180] [15401.830850] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15401.855949] LNet: Added LNI 192.168.201.116@tcp43 [8/256/0/180] [15410.594611] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15410.652451] LNet: Added LNI 192.168.201.116@tcp44 [8/256/0/180] [15420.458149] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15420.507251] LNet: Added LNI 192.168.201.116@tcp45 [8/256/0/180] [15429.033028] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15429.079647] LNet: Added LNI 192.168.201.116@tcp46 [8/256/0/180] [15439.378755] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15439.422772] LNet: Added LNI 192.168.201.116@tcp47 [8/256/0/180] [15450.414227] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15450.433475] LNet: Added LNI 192.168.201.116@tcp48 [8/256/0/180] [15459.215647] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15459.247673] LNet: Added LNI 192.168.201.116@tcp49 [8/256/0/180] [15468.153203] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15468.187384] LNet: Added LNI 192.168.201.116@tcp50 [8/256/0/180] [15488.992345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15488.997100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15489.008578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15490.529729] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15490.531457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15490.540958] Lustre: Skipped 1 previous similar message [15490.547227] Lustre: Skipped 1 previous similar message [15493.067075] Lustre: server umount lustre-MDT0000 complete [15496.353118] LustreError: 473053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855962 with bad export cookie 992419700887556910 [15496.356242] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15496.375809] LustreError: 473053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15496.737938] Lustre: server umount lustre-MDT0001 complete [15515.911636] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [15516.128271] Lustre: 472333:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855966/real 1787855966] req@000000004e7cfe58 x1874702436055424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15516.173332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15517.221814] Lustre: 472333:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855967/real 1787855967] req@000000001b9b8730 x1874702436055552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15521.247178] Lustre: 472331:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855971/real 1787855971] req@000000007eb56aa6 x1874702436055680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15525.957193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15528.416633] Lustre: 472334:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787855978/real 1787855978] req@00000000624b7fab x1874702436056064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787855994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15528.470790] Lustre: 472334:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15541.791648] LustreError: 472330:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000895b9ae1 x1874702436056704/t0(0) o250->MGC192.168.201.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15542.337133] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15542.385020] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15542.473583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15545.520765] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15545.533640] LustreError: 475026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15547.061308] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15551.976938] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15552.009900] LustreError: 475025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15556.008195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15556.372775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15561.543518] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15561.701730] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15568.802321] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15574.159813] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15593.962513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15593.971885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15593.988056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15593.997922] Lustre: Skipped 1 previous similar message [15597.536483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15597.557881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15597.562972] Lustre: Skipped 1 previous similar message [15599.851469] Lustre: server umount lustre-OST0000 complete [15607.779791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15607.781483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15607.782292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15607.805655] Lustre: Skipped 2 previous similar messages [15609.187217] Lustre: server umount lustre-MDT0000 complete [15612.144858] LustreError: 473053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856078 with bad export cookie 992419700887559507 [15612.149295] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15612.162771] LustreError: 473053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15612.509910] Lustre: server umount lustre-MDT0001 complete [15624.275072] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [15627.326594] Key type lgssc unregistered [15627.826476] LNet: 502053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15627.837932] LNetError: 502053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15627.850850] LNet: Removed LNI 192.168.201.116@tcp [15627.854778] LNet: Removed LNI 192.168.201.116@tcp10 [15627.868975] LNet: Removed LNI 192.168.201.116@tcp11 [15627.873541] LNet: Removed LNI 192.168.201.116@tcp12 [15627.877588] LNet: Removed LNI 192.168.201.116@tcp13 [15627.879922] LNet: Removed LNI 192.168.201.116@tcp14 [15627.882679] LNet: Removed LNI 192.168.201.116@tcp15 [15627.885289] LNet: Removed LNI 192.168.201.116@tcp16 [15627.887529] LNet: Removed LNI 192.168.201.116@tcp17 [15627.891414] LNet: Removed LNI 192.168.201.116@tcp18 [15627.895490] LNet: Removed LNI 192.168.201.116@tcp19 [15627.900348] LNet: Removed LNI 192.168.201.116@tcp20 [15627.903606] LNet: Removed LNI 192.168.201.116@tcp21 [15627.906783] LNet: Removed LNI 192.168.201.116@tcp22 [15627.909761] LNet: Removed LNI 192.168.201.116@tcp23 [15627.912573] LNet: Removed LNI 192.168.201.116@tcp24 [15627.915708] LNet: Removed LNI 192.168.201.116@tcp25 [15627.918880] LNet: Removed LNI 192.168.201.116@tcp26 [15627.922218] LNet: Removed LNI 192.168.201.116@tcp27 [15627.925432] LNet: Removed LNI 192.168.201.116@tcp28 [15627.928494] LNet: Removed LNI 192.168.201.116@tcp29 [15627.932452] LNet: Removed LNI 192.168.201.116@tcp30 [15627.935809] LNet: Removed LNI 192.168.201.116@tcp31 [15627.938620] LNet: Removed LNI 192.168.201.116@tcp32 [15627.940804] LNet: Removed LNI 192.168.201.116@tcp33 [15627.943730] LNet: Removed LNI 192.168.201.116@tcp34 [15627.948220] LNet: Removed LNI 192.168.201.116@tcp35 [15627.951624] LNet: Removed LNI 192.168.201.116@tcp36 [15627.954682] LNet: Removed LNI 192.168.201.116@tcp37 [15627.957994] LNet: Removed LNI 192.168.201.116@tcp38 [15627.960509] LNet: Removed LNI 192.168.201.116@tcp39 [15627.963665] LNet: Removed LNI 192.168.201.116@tcp40 [15627.966632] LNet: Removed LNI 192.168.201.116@tcp41 [15627.969569] LNet: Removed LNI 192.168.201.116@tcp42 [15627.972518] LNet: Removed LNI 192.168.201.116@tcp43 [15627.974921] LNet: Removed LNI 192.168.201.116@tcp44 [15627.976966] LNet: Removed LNI 192.168.201.116@tcp45 [15627.979889] LNet: Removed LNI 192.168.201.116@tcp46 [15627.982768] LNet: Removed LNI 192.168.201.116@tcp47 [15627.985764] LNet: Removed LNI 192.168.201.116@tcp48 [15627.987968] LNet: Removed LNI 192.168.201.116@tcp49 [15627.998411] LNet: Removed LNI 192.168.201.116@tcp50 [15628.683825] Key type .llcrypt unregistered [15628.688714] Key type ._llcrypt unregistered [15649.319076] Key type ._llcrypt registered [15649.320737] Key type .llcrypt registered [15649.453793] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [15664.909907] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [15666.800704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15666.871490] alg: No test for adler32 (adler32-zlib) [15667.950391] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [15668.155120] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [15669.840731] Key type lgssc registered [15670.956812] Lustre: Echo OBD driver; http://www.lustre.org/ [15708.506198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15708.581321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15709.918338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15709.940533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15710.053311] Lustre: lustre-MDT0000: new disk, initializing [15710.184935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15710.213921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15714.967918] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15726.757616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15726.883299] Lustre: 505247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15726.936863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15726.939673] Lustre: Skipped 1 previous similar message [15727.008578] Lustre: lustre-MDT0001: new disk, initializing [15727.110581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15727.140642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15727.152886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15732.144951] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15742.451614] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15748.966557] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15757.547072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15757.738441] Lustre: *** cfs_fail_loc=90f, val=0*** [15757.789951] Lustre: lustre-OST0000: new disk, initializing [15757.798322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15757.803084] Lustre: 507265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15757.848851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15765.072300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15765.101674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15765.235124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15765.576353] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15774.976091] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15785.956728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15785.964592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15785.970840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15791.073750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15791.086961] Lustre: Skipped 2 previous similar messages [15796.209642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15796.220082] Lustre: Skipped 1 previous similar message [15797.215160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15797.375830] Lustre: server umount lustre-OST0000 complete [15803.877164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15803.887210] LustreError: Skipped 1 previous similar message [15803.889622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15803.912548] Lustre: Skipped 1 previous similar message [15803.920051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15806.436638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15808.651950] Lustre: server umount lustre-MDT0000 complete [15813.049281] LustreError: 505241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856279 with bad export cookie 14233269688958102679 [15813.055740] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15813.059159] LustreError: 505241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15813.585293] Lustre: server umount lustre-MDT0001 complete [15824.253734] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [15827.034721] Key type lgssc unregistered [15827.360446] LNet: 509170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15827.378860] LNetError: 509170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15827.403797] LNet: Removed LNI 192.168.201.116@tcp [15828.391287] Key type .llcrypt unregistered [15828.397423] Key type ._llcrypt unregistered [15848.298522] Key type ._llcrypt registered [15848.301692] Key type .llcrypt registered [15848.441183] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [15849.493517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15849.514497] alg: No test for adler32 (adler32-zlib) [15850.692018] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [15851.032523] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [15852.743196] Key type lgssc registered [15854.064367] Lustre: Echo OBD driver; http://www.lustre.org/ [15861.297168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15861.318041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15862.733730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15866.938971] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15872.996953] LustreError: 510440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15873.034691] LustreError: 510440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15876.612707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15877.067746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15881.302440] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15889.273567] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15894.698228] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15902.289945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15902.578664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15909.509798] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15917.181751] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15928.933963] loop0: detected capacity change from 0 to 204800000 [15929.128498] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15929.152800] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15929.178059] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15935.556440] loop0: detected capacity change from 0 to 204800000 [15935.820087] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15935.828567] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15935.839543] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15941.463902] loop0: detected capacity change from 0 to 204800000 [15947.020427] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15947.404174] Lustre: fs15246-OST0000: new disk, initializing [15947.413785] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15947.431658] Lustre: 514033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15947.585821] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15954.899539] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15963.606694] loop1: detected capacity change from 0 to 204800000 [15969.173143] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15969.289732] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15969.339056] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15969.428234] Lustre: fs15246-MDT0000: new disk, initializing [15969.567755] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15969.593993] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15969.696546] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15970.789456] Lustre: 515155:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005f8d1a5f x1874703225501440/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787856452 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15973.306261] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [15975.846193] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15975.852429] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15975.907985] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16000.999585] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16001.043237] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16003.026799] Lustre: server umount fs15246-MDT0000 complete [16006.119173] LustreError: 510436:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16011.243427] LustreError: 510436:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16014.810191] Lustre: server umount fs15246-OST0000 complete [16031.715247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16031.715592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16031.719749] LustreError: Skipped 1 previous similar message [16031.733667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16031.739802] Lustre: Skipped 1 previous similar message [16031.748932] Lustre: Skipped 1 previous similar message [16035.310692] Lustre: server umount lustre-OST0000 complete [16040.929896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16040.952904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16040.982393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16046.971289] Lustre: server umount lustre-MDT0000 complete [16047.072225] LustreError: 510440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16047.086908] LustreError: 510440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16051.171863] LustreError: 510439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16051.209768] LustreError: 510439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16052.536753] LustreError: 510419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856518 with bad export cookie 1387987325674962762 [16052.546520] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16052.556473] LustreError: 510419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16052.826693] Lustre: server umount lustre-MDT0001 complete [16065.285716] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [16068.774940] Key type lgssc unregistered [16069.100924] LNet: 518254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16069.108273] LNetError: 518254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16069.123390] LNet: Removed LNI 192.168.201.116@tcp [16069.881149] Key type .llcrypt unregistered [16069.885494] Key type ._llcrypt unregistered [16089.770472] Key type ._llcrypt registered [16089.772831] Key type .llcrypt registered [16089.897340] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [16090.883484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16090.910436] alg: No test for adler32 (adler32-zlib) [16092.255931] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [16092.541455] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [16094.255934] Key type lgssc registered [16095.465794] Lustre: Echo OBD driver; http://www.lustre.org/ [16133.031060] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [16145.313309] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [16186.223560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16186.241395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16187.552379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16187.589674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16187.706937] Lustre: lustre-MDT0000: new disk, initializing [16187.847925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16187.864930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16191.872748] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [16201.793031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16201.935400] Lustre: 522718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16201.976763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16201.980789] Lustre: Skipped 1 previous similar message [16202.077767] Lustre: lustre-MDT0001: new disk, initializing [16202.204966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16202.292132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16202.312530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16206.476643] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [16215.377693] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16221.603625] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16229.141068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16229.347631] Lustre: lustre-OST0000: new disk, initializing [16229.351053] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16229.355669] Lustre: 524683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16229.438579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16236.076623] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [16236.642831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16236.660183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16236.805363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16248.590535] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16255.675126] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16255.999542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16262.530649] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16262.885688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16267.745270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16267.749910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16267.774488] LustreError: Skipped 1 previous similar message [16267.787396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16267.797502] Lustre: Skipped 1 previous similar message [16267.817544] Lustre: Skipped 1 previous similar message [16271.826356] Lustre: server umount lustre-OST0000 complete [16277.984293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16277.987143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16278.002726] Lustre: Skipped 1 previous similar message [16283.104852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16283.112430] Lustre: Skipped 2 previous similar messages [16287.202439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16287.205410] Lustre: Skipped 1 previous similar message [16290.271177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16290.573779] Lustre: server umount lustre-MDT0000 complete [16294.902200] LustreError: 522708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856760 with bad export cookie 14587229732331964282 [16294.904364] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16294.910782] LustreError: 522708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16295.282353] Lustre: server umount lustre-MDT0001 complete [16319.336758] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [16329.854642] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [16359.668513] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16014 sec ========= 14:53:43 (1787856823) [16361.406089] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:53:45 (1787856825) === [16364.891684] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:53:48 (1787856828) === [16404.870978] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [16407.159521] Key type lgssc unregistered [16407.419933] LNet: 531330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16407.435032] LNetError: 531330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16407.454985] LNet: Removed LNI 192.168.201.116@tcp [16408.495613] Key type .llcrypt unregistered [16408.497482] Key type ._llcrypt unregistered