[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514635675 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003175] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.011131] LSM: Security Framework initializing [ 0.012068] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022801] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027676] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030144] *** VALIDATE tmpfs *** [ 0.032409] *** VALIDATE proc *** [ 0.033286] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036103] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043840] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.046189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048033] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050017] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055324] rcu: Hierarchical SRCU implementation. [ 0.057545] smp: Bringing up secondary CPUs ... [ 0.058626] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.062409] smp: Brought up 1 node, 4 CPUs [ 0.064019] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140642] node 0 deferred pages initialised in 72ms [ 0.143054] devtmpfs: initialized [ 0.144227] x86/mm: Memory block size: 128MB [ 0.146364] gcov: version magic: 0x41383552 [ 0.149312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151317] pinctrl core: initialized pinctrl subsystem [ 0.152200] [ 0.152766] ************************************************************* [ 0.153015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154015] ** ** [ 0.155027] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156013] ** ** [ 0.157017] ** This means that this kernel is built to expose internal ** [ 0.158028] ** IOMMU data structures, which may compromise security on ** [ 0.159010] ** your system. ** [ 0.160015] ** ** [ 0.161017] ** If you see this message and you are not debugging the ** [ 0.162013] ** kernel, report this immediately to your vendor! ** [ 0.163014] ** ** [ 0.164018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ************************************************************* [ 0.166725] NET: Registered protocol family 16 [ 0.167492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170511] cpuidle: using governor menu [ 0.171000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173371] PCI: Using configuration type 1 for base access [ 0.174095] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186077] cryptd: max_cpu_qlen set to 1000 [ 0.188252] ACPI: Added _OSI(Module Device) [ 0.189014] ACPI: Added _OSI(Processor Device) [ 0.190013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190872] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192873] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194423] ACPI: Interpreter enabled [ 0.195048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195902] ACPI: Using IOAPIC for interrupt routing [ 0.196072] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197283] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.204223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.205042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.207073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.208946] acpiphp: Slot [2] registered [ 0.209061] acpiphp: Slot [5] registered [ 0.209884] acpiphp: Slot [6] registered [ 0.210059] acpiphp: Slot [7] registered [ 0.210794] acpiphp: Slot [8] registered [ 0.211065] acpiphp: Slot [9] registered [ 0.212091] acpiphp: Slot [10] registered [ 0.212953] acpiphp: Slot [3] registered [ 0.213051] acpiphp: Slot [4] registered [ 0.214113] acpiphp: Slot [11] registered [ 0.214979] acpiphp: Slot [12] registered [ 0.215101] acpiphp: Slot [13] registered [ 0.216071] acpiphp: Slot [14] registered [ 0.216867] acpiphp: Slot [15] registered [ 0.217049] acpiphp: Slot [16] registered [ 0.217802] acpiphp: Slot [17] registered [ 0.218051] acpiphp: Slot [18] registered [ 0.218689] acpiphp: Slot [19] registered [ 0.219095] acpiphp: Slot [20] registered [ 0.220036] acpiphp: Slot [21] registered [ 0.221073] acpiphp: Slot [22] registered [ 0.222079] acpiphp: Slot [23] registered [ 0.223092] acpiphp: Slot [24] registered [ 0.224145] acpiphp: Slot [25] registered [ 0.225000] acpiphp: Slot [26] registered [ 0.226084] acpiphp: Slot [27] registered [ 0.227058] acpiphp: Slot [28] registered [ 0.228000] acpiphp: Slot [29] registered [ 0.230097] acpiphp: Slot [30] registered [ 0.231102] acpiphp: Slot [31] registered [ 0.231977] PCI host bridge to bus 0000:00 [ 0.233018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.236012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.238076] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.240042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.242023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.243164] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.245695] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.248000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.255015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.258086] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.261014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.264014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.265463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.267563] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.269025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.272668] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.275908] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.285016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.289010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.292846] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.298019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.304028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.317019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.328093] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.333019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.337018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.349014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.356000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.361021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.368012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.379012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.388830] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.394017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.400022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.414024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.420908] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.427017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.431018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.444016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.453728] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.460015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.465024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.478015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.486714] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.489251] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.490276] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.492308] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.493138] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.498029] iommu: Default domain type: Passthrough [ 0.499450] SCSI subsystem initialized [ 0.501155] ACPI: bus type USB registered [ 0.503104] usbcore: registered new interface driver usbfs [ 0.505080] usbcore: registered new interface driver hub [ 0.507103] usbcore: registered new device driver usb [ 0.509171] pps_core: LinuxPPS API ver. 1 registered [ 0.511012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.514066] PTP clock support registered [ 0.515189] EDAC MC: Ver: 3.0.0 [ 0.517129] PCI: Using ACPI for IRQ routing [ 0.518894] NetLabel: Initializing [ 0.520018] NetLabel: domain hash size = 128 [ 0.522011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.524090] NetLabel: unlabeled traffic allowed by default [ 0.526090] vgaarb: loaded [ 0.527180] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.529084] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.534520] clocksource: Switched to clocksource kvm-clock [ 0.649732] VFS: Disk quotas dquot_6.6.0 [ 0.651671] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.654676] *** VALIDATE ramfs *** [ 0.656158] *** VALIDATE hugetlbfs *** [ 0.658085] pnp: PnP ACPI init [ 0.661686] pnp: PnP ACPI: found 6 devices [ 0.679889] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.683796] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.685407] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.686771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.689212] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.691497] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.694504] NET: Registered protocol family 2 [ 0.697301] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.701861] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.704629] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.709252] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.712708] TCP: Hash tables configured (established 65536 bind 65536) [ 0.715512] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.718587] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.721334] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.724418] NET: Registered protocol family 1 [ 0.726932] RPC: Registered named UNIX socket transport module. [ 0.728884] RPC: Registered udp transport module. [ 0.730437] RPC: Registered tcp transport module. [ 0.732046] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.734272] NET: Registered protocol family 44 [ 0.735819] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.737845] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.739804] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.742094] PCI: CLS 0 bytes, default 64 [ 0.743768] Unpacking initramfs... [ 2.145552] debug: unmapping init [mem 0xffff8e11bcc54000-0xffff8e11bffbffff] [ 2.149367] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.151748] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.154938] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.671991] Initialise system trusted keyrings [ 2.673720] Key type blacklist registered [ 2.675588] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.685873] zbud: loaded [ 2.690245] *** VALIDATE nfs *** [ 2.691621] *** VALIDATE nfs4 *** [ 2.694118] pstore: using deflate compression [ 2.698024] Platform Keyring initialized [ 2.833845] NET: Registered protocol family 38 [ 2.836204] Key type asymmetric registered [ 2.837745] Asymmetric key parser 'x509' registered [ 2.839061] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.842287] io scheduler mq-deadline registered [ 2.843822] io scheduler kyber registered [ 2.845572] io scheduler bfq registered [ 2.847924] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.850968] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.853789] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.856767] ACPI: Power Button [PWRF] [ 2.864254] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.872591] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.888787] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.897712] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.909025] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.935400] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.965360] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.969981] Non-volatile memory driver v1.3 [ 2.971314] Linux agpgart interface v0.103 [ 2.999608] virtio_blk virtio1: [vda] 146856 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.001978] vda: detected capacity change from 0 to 75190272 [ 3.015767] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.017771] vdb: detected capacity change from 0 to 1073741824 [ 3.029833] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.032064] vdc: detected capacity change from 0 to 2621440000 [ 3.043671] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.045775] vdd: detected capacity change from 0 to 2621440000 [ 3.056555] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.058423] vde: detected capacity change from 0 to 4294967296 [ 3.071214] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.073413] vdf: detected capacity change from 0 to 4294967296 [ 3.079111] libphy: Fixed MDIO Bus: probed [ 3.085488] usbcore: registered new interface driver usbserial_generic [ 3.087385] usbserial: USB Serial support registered for generic [ 3.089128] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.092934] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.094384] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.096194] mousedev: PS/2 mouse device common for all mice [ 3.099358] rtc_cmos 00:05: RTC can wake from S4 [ 3.101747] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.106092] rtc_cmos 00:05: registered as rtc0 [ 3.107845] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.108464] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.114641] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.114692] intel_pstate: CPU model not supported [ 3.120826] hid: raw HID events driver (C) Jiri Kosina [ 3.122723] usbcore: registered new interface driver usbhid [ 3.124939] usbhid: USB HID core driver [ 3.126264] drop_monitor: Initializing network drop monitor service [ 3.128170] Initializing XFRM netlink socket [ 3.129860] NET: Registered protocol family 10 [ 3.132341] Segment Routing with IPv6 [ 3.133553] NET: Registered protocol family 17 [ 3.135219] mpls_gso: MPLS GSO support [ 3.141445] RAS: Correctable Errors collector initialized. [ 3.143469] AVX version of gcm_enc/dec engaged. [ 3.144630] AES CTR mode by8 optimization enabled [ 3.217075] sched_clock: Marking stable (3217028287, 0)->(4145422851, -928394564) [ 3.220292] registered taskstats version 1 [ 3.222837] Loading compiled-in X.509 certificates [ 3.225107] zswap: loaded using pool lzo/zbud [ 3.249636] Key type big_key registered [ 3.263435] Key type encrypted registered [ 3.264746] ima: No TPM chip found, activating TPM-bypass! [ 3.266663] ima: Allocated hash algorithm: sha1 [ 3.268350] ima: No architecture policies found [ 3.269953] evm: Initialising EVM extended attributes: [ 3.271886] evm: security.selinux [ 3.273145] evm: security.ima [ 3.274172] evm: security.capability [ 3.275655] evm: HMAC attrs: 0x1 [ 3.278066] rtc_cmos 00:05: setting system clock to 2026-08-27 09:36:31 UTC (1787823391) [ 3.284422] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.287025] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 3.296096] Write protecting the kernel read-only data: 28672k [ 3.299300] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.301915] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 3.336475] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.342628] systemd[1]: Detected virtualization kvm. [ 3.343602] systemd[1]: Detected architecture x86-64. [ 3.344836] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.371328] systemd[1]: No hostname configured. [ 3.372990] systemd[1]: Set hostname to . [ 3.374845] random: systemd: uninitialized urandom read (16 bytes read) [ 3.376565] systemd[1]: Initializing machine ID from random generator. [ 3.419943] random: ln: uninitialized urandom read (6 bytes read) [ 3.491149] random: systemd: uninitialized urandom read (16 bytes read) [ 3.493832] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.497990] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.501942] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.994817] device-mapper: uevent: version 1.0.3 [ 3.996639] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.611163] random: fast init done [ 4.613768] virtio_net virtio0 ens2: renamed from eth0 [ 4.628098] scsi host0: ata_piix [ 4.690220] scsi host1: ata_piix [ 4.709198] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.711551] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.314136] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.781142] random: crng init done [ 9.782622] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.006748] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.062330] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.288973] SELinux: Disabled at runtime. [ 11.348467] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.356766] systemd[1]: Detected virtualization kvm. [ 11.358447] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.767104] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.770485] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.774891] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.778759] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.782049] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.789330] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.793907] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ 11.881070] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.228096] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.459773] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.529435] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.692246] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.701632] EDAC sbridge: Ver: 1.1.2 [ 14.267626] Key type dns_resolver registered [ 14.568255] NFS: Registering the id_resolver key type [ 14.570240] Key type id_resolver registered [ 14.571836] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-server login: [ 75.120856] libcfs: loading out-of-tree module taints kernel. [ 75.182563] Key type ._llcrypt registered [ 75.185861] Key type .llcrypt registered [ 75.293720] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 92.207368] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 93.512529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.525801] alg: No test for adler32 (adler32-zlib) [ 95.145489] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 96.329491] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 98.272150] Key type lgssc registered [ 100.335639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.066965] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 167.549951] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 182.930613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.975383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.342488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.372504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.500899] Lustre: lustre-MDT0000: new disk, initializing [ 184.590428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.624811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.373635] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.354015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.512565] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.569558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.575318] Lustre: Skipped 1 previous similar message [ 203.739984] Lustre: lustre-MDT0001: new disk, initializing [ 203.853663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.907844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.924886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.215785] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.574548] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.409685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.810239] Lustre: lustre-OST0000: new disk, initializing [ 224.819447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.828192] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.938985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.430493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.448087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.555027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.668888] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.989454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.218303] Lustre: lustre-OST0001: new disk, initializing [ 250.227421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 250.247681] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.347891] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.513016] hrtimer: interrupt took 3503726 ns [ 258.068589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.071396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.105115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 258.114524] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.371534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.263600] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.232243] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.822744] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing yml_node [ 297.069132] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 299.868630] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 302.597571] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 304.341737] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 05:41:31 EDT 2026 [ 323.329604] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 325.249341] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.057300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.061202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.082142] Lustre: Skipped 1 previous similar message [ 345.109024] Lustre: Skipped 3 previous similar messages [ 348.888993] Lustre: server umount lustre-MDT0000 complete [ 353.264151] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823741 with bad export cookie 1333341288125914869 [ 353.264481] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 353.277328] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 355.306442] LustreError: 10055:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.307063] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.310233] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.317470] LustreError: 10055:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 355.354424] Lustre: Skipped 3 previous similar messages [ 358.305086] LustreError: 12152:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.318739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 358.340268] LustreError: 12152:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 358.373417] Lustre: Skipped 2 previous similar messages [ 359.853741] Lustre: server umount lustre-MDT0001 complete [ 364.473193] Lustre: server umount lustre-OST0000 complete [ 368.767702] Lustre: server umount lustre-OST0001 complete [ 376.323655] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 387.297282] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 427.709464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.029868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.078512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.155468] Lustre: lustre-MDT0000: new disk, initializing [ 428.304684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.335620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 434.117802] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 446.334396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.524607] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 446.633320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 446.647669] Lustre: Skipped 1 previous similar message [ 446.775615] Lustre: lustre-MDT0001: new disk, initializing [ 446.953410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.047737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.063510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 452.475358] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 462.004634] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.534583] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 476.028512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.400327] Lustre: lustre-OST0000: new disk, initializing [ 476.412517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 476.427649] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 476.563462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.425284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 478.436830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 478.569040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 483.340292] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 494.309419] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.957790] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 500.222181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 505.279541] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 505.619755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.416965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.419859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.437841] LustreError: Skipped 1 previous similar message [ 509.441216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.481389] Lustre: Skipped 1 previous similar message [ 513.806944] Lustre: server umount lustre-OST0000 complete [ 518.630515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.639311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.661357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.670935] Lustre: Skipped 1 previous similar message [ 524.661692] Lustre: server umount lustre-MDT0000 complete [ 524.775133] LustreError: 15084:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.791295] LustreError: 15084:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 528.770734] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823916 with bad export cookie 1333341288125916402 [ 528.806407] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.431927] Lustre: server umount lustre-MDT0001 complete [ 533.706799] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:45:21 (1787823921) [ 536.594582] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 538.564160] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:45:25 (1787823925) [ 545.164251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.528584] LustreError: 19457:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.608302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.925127] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 550.889415] LustreError: 19458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.425609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.955864] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 568.919835] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 573.668797] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.479232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 586.891919] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 595.790090] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 604.651335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 604.665782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.678929] Lustre: Skipped 1 previous similar message [ 604.687815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 604.694948] Lustre: Skipped 2 previous similar messages [ 610.013498] Lustre: server umount lustre-OST0000 complete [ 614.888101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.901478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.925345] Lustre: Skipped 1 previous similar message [ 621.028036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 621.047973] Lustre: Skipped 6 previous similar messages [ 628.192115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 628.368589] Lustre: server umount lustre-MDT0000 complete [ 631.266970] LustreError: 20185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.305526] LustreError: 20185:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 632.193947] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824020 with bad export cookie 1333341288125917284 [ 632.215376] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 632.225843] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.582772] Lustre: server umount lustre-MDT0001 complete [ 644.234947] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 647.533180] Key type lgssc unregistered [ 647.820187] LNet: 22991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.831753] LNetError: 22991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.872649] LNet: Removed LNI 192.168.201.125@tcp [ 649.743194] Key type .llcrypt unregistered [ 649.746448] Key type ._llcrypt unregistered [ 672.246794] Key type ._llcrypt registered [ 672.252372] Key type .llcrypt registered [ 672.353472] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 673.168119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 673.199569] alg: No test for adler32 (adler32-zlib) [ 674.226172] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 674.431286] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 676.112328] Key type lgssc registered [ 677.553940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.884402] loop: module loaded [ 684.962166] loop0: detected capacity change from 0 to 204800000 [ 685.236879] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.255794] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.686453] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.587681] loop0: detected capacity change from 0 to 204800000 [ 693.896952] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.909501] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.936293] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.970930] loop0: detected capacity change from 0 to 204800000 [ 711.098775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 711.128934] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.370577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 712.399273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 712.456935] Lustre: lustre-MDT0000: new disk, initializing [ 712.535396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.566487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 719.977753] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 728.127192] loop1: detected capacity change from 0 to 204800000 [ 733.125317] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.354390] Lustre: lustre-OST07c6: new disk, initializing [ 733.357496] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 733.360625] Lustre: Skipped 1 previous similar message [ 733.364313] Lustre: 26179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 733.419531] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 739.586644] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 739.988549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 739.995798] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 740.026165] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 744.949078] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 744.958876] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 744.969797] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 744.981840] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 744.992337] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 745.002876] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 757.717263] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 757.725496] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 757.729814] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.734815] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 757.743419] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 757.749388] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.469322] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 762.482472] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.492475] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 762.498103] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.508866] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.513484] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.521613] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 762.528947] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.533647] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.538582] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.548656] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.557568] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.742536] Lustre: 26697:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 765.756257] Lustre: 26697:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 765.770765] Lustre: 26697:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 765.783240] Lustre: 26697:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.797274] Lustre: 26697:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.812741] Lustre: 26697:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.828926] Lustre: 26697:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 765.837394] Lustre: 26697:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.854677] Lustre: 26697:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 765.871689] Lustre: 26697:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.879508] Lustre: 26697:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.892569] Lustre: 26697:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.553910] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 770.566279] Lustre: 25163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 770.573068] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 770.578671] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.583503] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.592661] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.604923] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 770.616416] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.632747] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.641218] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.652319] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.662621] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.264220] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 781.288078] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 781.300283] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 781.308178] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 781.316850] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.333693] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 781.353865] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 781.371407] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 781.380353] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.389012] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 781.399833] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.409060] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 827.768938] Lustre: server umount lustre-OST07c6 complete [ 846.017794] Lustre: server umount lustre-MDT0000 complete [ 861.374607] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 864.111697] Key type lgssc unregistered [ 864.454619] LNet: 28735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.465641] LNetError: 28735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.486927] LNet: Removed LNI 192.168.201.125@tcp [ 865.525974] Key type .llcrypt unregistered [ 865.528688] Key type ._llcrypt unregistered [ 884.980723] Key type ._llcrypt registered [ 884.982323] Key type .llcrypt registered [ 885.068354] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 886.203472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.497074] alg: No test for adler32 (adler32-zlib) [ 887.540866] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 887.845890] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 889.568185] Key type lgssc registered [ 890.699438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 897.840738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 897.855313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.283693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.268636] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 909.798913] LustreError: 30006:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.811489] LustreError: 30006:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 912.725531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.177624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.903943] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 923.649380] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 929.457830] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 936.651421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.163789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.486828] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 945.489778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 950.130489] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 969.190728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.206559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.223627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.233912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 971.238533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.620281] Lustre: server umount lustre-OST0000 complete [ 979.937493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 979.945704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.970495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.982399] Lustre: Skipped 1 previous similar message [ 984.123959] Lustre: server umount lustre-MDT0000 complete [ 986.606117] LustreError: 31356:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.638796] LustreError: 31356:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 988.357428] LustreError: 29987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824376 with bad export cookie 6050751957852824001 [ 988.359503] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.371290] LustreError: 29987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 988.750237] Lustre: server umount lustre-MDT0001 complete [ 1000.892371] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1004.049966] Key type lgssc unregistered [ 1004.368103] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.392877] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.434193] LNet: Removed LNI 192.168.201.125@tcp [ 1005.341969] Key type .llcrypt unregistered [ 1005.344130] Key type ._llcrypt unregistered [ 1024.066777] Key type ._llcrypt registered [ 1024.070912] Key type .llcrypt registered [ 1024.170570] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1025.494769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.517951] alg: No test for adler32 (adler32-zlib) [ 1026.686553] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1026.938791] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1028.672240] Key type lgssc registered [ 1030.285485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.826191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.838661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.268674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.593892] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1050.607514] LustreError: 35020:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.634222] LustreError: 35020:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1053.918105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.171698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1058.987302] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1066.136165] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1072.212284] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.484044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.795345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.973068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1085.223465] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1093.577720] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1105.414900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1105.416302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.423919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.428454] LustreError: Skipped 1 previous similar message [ 1105.450372] Lustre: Skipped 1 previous similar message [ 1110.501368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.516469] Lustre: Skipped 1 previous similar message [ 1111.136683] Lustre: server umount lustre-OST0000 complete [ 1115.627360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1115.630830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.686240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.693939] Lustre: Skipped 2 previous similar messages [ 1120.738109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.751166] Lustre: Skipped 1 previous similar message [ 1121.870976] Lustre: server umount lustre-MDT0000 complete [ 1125.862300] LustreError: 35015:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.883498] LustreError: 35015:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1126.139148] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824514 with bad export cookie 6871043279527466138 [ 1126.152216] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.555858] Lustre: server umount lustre-MDT0001 complete [ 1139.932737] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1143.171316] Key type lgssc unregistered [ 1143.557258] LNet: 38553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.562585] LNetError: 38553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.574622] LNet: Removed LNI 192.168.201.125@tcp [ 1144.397951] Key type .llcrypt unregistered [ 1144.399992] Key type ._llcrypt unregistered [ 1168.322678] Key type ._llcrypt registered [ 1168.326428] Key type .llcrypt registered [ 1168.472938] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1169.802342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1169.866905] alg: No test for adler32 (adler32-zlib) [ 1170.942803] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1171.323499] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1173.000212] Key type lgssc registered [ 1173.965121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1180.416465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1180.429595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.766977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.444289] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1190.885991] LustreError: 39970:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.904277] LustreError: 39970:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1194.992983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.326841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.213139] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1210.701625] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.757872] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.361181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.933646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.025705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1232.368714] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1242.623184] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1250.773518] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1251.811184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.818108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.852678] Lustre: Skipped 2 previous similar messages [ 1251.905550] Lustre: Skipped 2 previous similar messages [ 1255.809992] Lustre: server umount lustre-MDT0000 complete [ 1255.871532] LustreError: 39967:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.934188] LustreError: 39970:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.888396] LustreError: 39951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824648 with bad export cookie 10986391416734249945 [ 1259.904280] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.917710] LustreError: 39951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1260.976033] LustreError: 39966:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.002123] LustreError: 39966:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1261.027429] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [ 1262.064630] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.098071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1264.112671] Lustre: Skipped 1 previous similar message [ 1266.097762] LustreError: 41887:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.139088] LustreError: 41887:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1269.219472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1269.232820] Lustre: Skipped 1 previous similar message [ 1274.336465] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1274.629619] Lustre: server umount lustre-MDT0001 complete [ 1290.081646] Lustre: server umount lustre-OST0000 complete [ 1306.830615] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1309.585183] Key type lgssc unregistered [ 1309.931879] LNet: 43838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.936825] LNetError: 43838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.950528] LNet: Removed LNI 192.168.201.125@tcp [ 1310.755858] Key type .llcrypt unregistered [ 1310.758678] Key type ._llcrypt unregistered [ 1336.675489] Key type ._llcrypt registered [ 1336.678209] Key type .llcrypt registered [ 1336.775475] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1337.896682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.092227] alg: No test for adler32 (adler32-zlib) [ 1339.112330] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1339.343762] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1341.032194] Key type lgssc registered [ 1342.282119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.557507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.590955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.092550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.796583] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1361.391788] LustreError: 45256:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.440804] LustreError: 45256:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1362.400267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.693053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1368.490813] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1377.284356] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1383.704832] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1391.478531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.989950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.096746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1398.972287] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1406.548599] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1414.634468] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1416.674214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.684601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.701461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.703211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1417.704228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.716353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.742019] Lustre: Skipped 1 previous similar message [ 1420.127891] Lustre: server umount lustre-OST0000 complete [ 1439.715050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.725167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.735676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.739587] Lustre: Skipped 1 previous similar message [ 1443.298342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.043538] Lustre: server umount lustre-MDT0000 complete [ 1448.421251] LustreError: 45252:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.456680] LustreError: 45252:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1450.306466] LustreError: 47100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824838 with bad export cookie 2248184523962043964 [ 1450.315196] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.286222] Lustre: server umount lustre-MDT0001 complete [ 1470.381545] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1474.362914] Key type lgssc unregistered [ 1474.773759] LNet: 48982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.781281] LNetError: 48982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.811863] LNet: Removed LNI 192.168.201.125@tcp [ 1475.845425] Key type .llcrypt unregistered [ 1475.847127] Key type ._llcrypt unregistered [ 1502.205458] Key type ._llcrypt registered [ 1502.211305] Key type .llcrypt registered [ 1502.378390] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1503.666321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.717614] alg: No test for adler32 (adler32-zlib) [ 1505.360680] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1505.776180] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1507.672309] Key type lgssc registered [ 1509.282190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1519.350752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1519.376696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.933990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.370756] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1531.365714] LustreError: 50399:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.399015] LustreError: 50399:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1535.662706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.068432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1540.567453] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1550.278247] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1557.191796] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1566.898333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1567.730437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.621415] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1582.395361] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1591.290418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1598.628548] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1604.150508] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1617.420416] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 6h03m23s [ 1619.050199] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1621.988184] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.990112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1621.998178] Lustre: Skipped 2 previous similar messages [ 1622.001574] Lustre: Skipped 1 previous similar message [ 1623.078477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.083223] Lustre: Skipped 3 previous similar messages [ 1625.077083] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1625.103978] Lustre: Skipped 1 previous similar message [ 1628.065533] Lustre: server umount lustre-MDT0000 complete [ 1629.163599] LustreError: 50396:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.202558] LustreError: 50396:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1631.150394] LustreError: 51133:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.216824] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1635.307276] LustreError: 50399:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.335787] LustreError: 50399:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1639.199524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.346740] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.661168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.743396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1640.687554] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1644.579238] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1648.268451] Lustre: DEBUG MARKER: Wait for df (36757) ... [ 1650.143927] Lustre: DEBUG MARKER: done [ 1655.266060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1655.272359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.291069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.317764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.412386] LustreError: 52844:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.432117] LustreError: 52844:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1661.638401] Lustre: server umount lustre-OST0000 complete [ 1666.554049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1666.555221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.569357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.569364] Lustre: Skipped 2 previous similar messages [ 1666.593157] Lustre: Skipped 1 previous similar message [ 1672.903940] Lustre: server umount lustre-MDT0000 complete [ 1676.887792] LustreError: 50379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825065 with bad export cookie 7705164979451773716 [ 1676.888626] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.903743] LustreError: 50379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1677.537106] Lustre: server umount lustre-MDT0001 complete [ 1688.202299] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1690.994328] Key type lgssc unregistered [ 1691.356508] LNet: 55359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.361845] LNetError: 55359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.376442] LNet: Removed LNI 192.168.201.125@tcp [ 1692.283188] Key type .llcrypt unregistered [ 1692.284926] Key type ._llcrypt unregistered [ 1732.974099] Key type ._llcrypt registered [ 1732.981531] Key type .llcrypt registered [ 1733.152339] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1734.690798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.759315] alg: No test for adler32 (adler32-zlib) [ 1735.915847] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1736.180131] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1737.896204] Key type lgssc registered [ 1739.380666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1747.933567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1747.962286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.389799] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1749.435603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1749.589531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.004725] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1764.493602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.643416] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1764.662866] Lustre: 57561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1764.920742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1770.241525] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1778.832279] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1785.734455] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1793.776039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1793.936212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1794.189154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.128365] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1804.531765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1809.757654] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1815.853395] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 6h06m43s [ 1817.319820] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1824.712730] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1827.110457] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1827.174273] LustreError: 59697:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1847.258667] Lustre: lustre-MDT0000: Client 2e3dd96f-d86b-4ab6-b22e-c19277284a70 (at 192.168.201.25@tcp) reconnecting [ 1867.176700] LustreError: 59697:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1870.137424] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1873.138675] Lustre: DEBUG MARKER: done [ 1881.586472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1881.586743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.601049] LustreError: Skipped 1 previous similar message [ 1881.610909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1881.628705] Lustre: Skipped 1 previous similar message [ 1881.650378] Lustre: Skipped 1 previous similar message [ 1886.557634] Lustre: server umount lustre-OST0000 complete [ 1891.042684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.048853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.053968] Lustre: Skipped 1 previous similar message [ 1891.067925] Lustre: Skipped 1 previous similar message [ 1892.130956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.137718] Lustre: Skipped 2 previous similar messages [ 1897.633208] Lustre: server umount lustre-MDT0000 complete [ 1902.041635] LustreError: 57552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825290 with bad export cookie 16563115543753931676 [ 1902.052994] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.056923] LustreError: 57552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1902.570314] Lustre: server umount lustre-MDT0001 complete [ 1916.823190] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1920.550528] Key type lgssc unregistered [ 1920.984758] LNet: 61897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.008533] LNetError: 61897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.022938] LNet: Removed LNI 192.168.201.125@tcp [ 1923.614084] Key type .llcrypt unregistered [ 1923.619367] Key type ._llcrypt unregistered [ 1979.231886] Key type ._llcrypt registered [ 1979.236350] Key type .llcrypt registered [ 1979.376327] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1980.908790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1981.000371] alg: No test for adler32 (adler32-zlib) [ 1982.070983] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1982.401791] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1984.192636] Key type lgssc registered [ 1985.696238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1989.345852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1989.385288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.783566] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1990.816481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1991.074577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.256208] Lustre: Failing over lustre-MDT0000 [ 1992.465568] Lustre: server umount lustre-MDT0000 complete [ 1999.982752] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:09:47 (1787825387) [ 2006.983236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.476659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.394464] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2021.817340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.978488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2022.017791] Lustre: 64705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2022.390522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.722929] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2036.199642] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2043.178808] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2050.412567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2050.583930] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2050.840759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.415405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2058.019481] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2065.799966] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2072.635511] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2083.309937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.320463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.324600] Lustre: Skipped 1 previous similar message [ 2083.345166] Lustre: Skipped 2 previous similar messages [ 2085.748263] Lustre: server umount lustre-MDT0000 complete [ 2089.572891] LustreError: 65439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825477 with bad export cookie 3187336555255721139 [ 2089.581357] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2089.590721] LustreError: 65439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2090.034658] Lustre: server umount lustre-MDT0001 complete [ 2093.983478] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2099.065867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2107.515663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.896353] Lustre: 63624:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787825481/real 1787825481] req@000000004e5cffc3 x1874670775216512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787825497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2108.914662] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.931756] Lustre: Skipped 1 previous similar message [ 2109.920183] Lustre: 63625:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787825482/real 1787825482] req@000000001c27c8eb x1874670775216640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787825498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.018615] Lustre: 63627:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787825486/real 1787825486] req@00000000d89024b5 x1874670775216768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787825502 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.081077] LustreError: 63623:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090e3570c x1874670775217536/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.317775] LustreError: 66553:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.340634] LustreError: 66553:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2114.419877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.512829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2118.443863] LustreError: 66554:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.483962] LustreError: 68473:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.519930] LustreError: 68473:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2119.599826] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2121.697160] Lustre: 63625:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787825493/real 1787825493] req@00000000e8039052 x1874670775217152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787825509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2121.746108] Lustre: 63625:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2124.778305] LustreError: 66554:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.800941] LustreError: 66554:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2128.628460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.193361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.063730] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2134.536322] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2138.627559] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2141.519470] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2149.324453] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.074409] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.093490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.110539] Lustre: Skipped 1 previous similar message [ 2159.133901] Lustre: Skipped 2 previous similar messages [ 2162.887977] Lustre: server umount lustre-MDT0000 complete [ 2164.202423] LustreError: 66554:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.224493] LustreError: 66554:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2166.951701] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825555 with bad export cookie 3187336555255723414 [ 2166.955531] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.968989] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.509526] Lustre: server umount lustre-MDT0001 complete [ 2173.923927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2184.488944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.696195] Lustre: 63627:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787825557/real 1787825557] req@0000000088872fd6 x1874670775244288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787825573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.728425] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.747282] Lustre: Skipped 1 previous similar message [ 2192.483342] LustreError: 69481:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.607771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.738296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2197.830561] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2199.009079] Lustre: 63626:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787825570/real 1787825570] req@000000002561d1e1 x1874670775244928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787825586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.040712] Lustre: 63626:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2207.539201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.375353] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2213.887949] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2221.105834] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.244771] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2234.853205] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.858861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2234.870201] Lustre: Skipped 1 previous similar message [ 2234.891269] Lustre: Skipped 2 previous similar messages [ 2239.970817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2239.986311] Lustre: Skipped 2 previous similar messages [ 2240.954615] Lustre: server umount lustre-MDT0000 complete [ 2244.379769] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825632 with bad export cookie 3187336555255724793 [ 2244.382078] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.393592] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2244.730968] Lustre: server umount lustre-MDT0001 complete [ 2251.414474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2255.555112] Lustre: DEBUG MARKER: files compared the same [ 2268.957573] Lustre: server umount lustre-OST0000 complete [ 2286.866029] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2290.042634] Key type lgssc unregistered [ 2290.397971] LNet: 74434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.416456] LNetError: 74434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.452153] LNet: Removed LNI 192.168.201.125@tcp [ 2291.781269] Key type .llcrypt unregistered [ 2291.787616] Key type ._llcrypt unregistered [ 2317.343506] Key type ._llcrypt registered [ 2317.346607] Key type .llcrypt registered [ 2317.445690] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2318.804177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.842804] alg: No test for adler32 (adler32-zlib) [ 2319.825267] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2320.064216] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2321.760189] Key type lgssc registered [ 2322.707689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2330.748590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2330.763788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.248390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.313941] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug +malloc all [ 2342.379493] LustreError: 75856:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.395091] LustreError: 75856:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2346.895742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.180227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2352.374170] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug +malloc all [ 2361.041748] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2368.589520] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2375.853174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2376.061840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.064817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2382.378312] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug +malloc all [ 2390.119930] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.788744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2402.790750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.839708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.845633] Lustre: Skipped 1 previous similar message [ 2405.660173] Lustre: server umount lustre-OST0000 complete [ 2413.032591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.037519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2413.042493] Lustre: Skipped 2 previous similar messages [ 2413.060631] Lustre: Skipped 1 previous similar message [ 2415.838977] Lustre: server umount lustre-MDT0000 complete [ 2418.154679] LustreError: 75850:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.178918] LustreError: 75850:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2420.498233] LustreError: 75837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825808 with bad export cookie 7898230007977284583 [ 2420.501273] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.513389] LustreError: 75837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2421.025942] Lustre: server umount lustre-MDT0001 complete [ 2433.751430] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2436.682544] Key type lgssc unregistered [ 2437.183760] LNet: 79389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.198321] LNetError: 79389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.236788] LNet: Removed LNI 192.168.201.125@tcp [ 2438.032716] Key type .llcrypt unregistered [ 2438.033845] Key type ._llcrypt unregistered [ 2465.941813] Key type ._llcrypt registered [ 2465.944667] Key type .llcrypt registered [ 2466.194085] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2468.240280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.303918] alg: No test for adler32 (adler32-zlib) [ 2469.530684] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2469.912437] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2471.673883] Key type lgssc registered [ 2473.047914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2482.212667] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2482.245528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.825699] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2502.945440] LustreError: 80782:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 2529.632237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.970793] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2542.942583] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2551.717681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.850615] Lustre: *** cfs_fail_loc=706, val=0*** [ 2552.313160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.430842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2557.422115] LustreError: 80808:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.441708] LustreError: 80808:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2557.814886] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2562.534862] LustreError: 80807:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.581888] LustreError: 81866:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.614291] LustreError: 81866:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2568.282430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.676292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2574.342850] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2583.002782] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2589.194453] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2593.249709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.262517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.281449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.278068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.281736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.291778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.590238] Lustre: server umount lustre-OST0000 complete [ 2604.515192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.517924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.521218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.521228] Lustre: Skipped 1 previous similar message [ 2604.535220] Lustre: Skipped 1 previous similar message [ 2608.320923] Lustre: server umount lustre-MDT0000 complete [ 2609.642960] LustreError: 81867:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.662284] LustreError: 81867:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2611.520680] LustreError: 80788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825999 with bad export cookie 16850981573410690419 [ 2611.534528] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.539486] LustreError: 80788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2611.897816] Lustre: server umount lustre-MDT0001 complete [ 2622.703537] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2626.204451] Key type lgssc unregistered [ 2626.556306] LNet: 84399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.577890] LNetError: 84399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.609314] LNet: Removed LNI 192.168.201.125@tcp [ 2627.472683] Key type .llcrypt unregistered [ 2627.475071] Key type ._llcrypt unregistered [ 2652.313558] Key type ._llcrypt registered [ 2652.318595] Key type .llcrypt registered [ 2652.444208] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2653.678523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2653.868298] alg: No test for adler32 (adler32-zlib) [ 2655.006485] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2655.320152] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2657.000215] Key type lgssc registered [ 2658.029689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.227272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.248957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.315850] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2678.108813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.579252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.380076] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2683.873955] LustreError: 86503:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.888573] LustreError: 86503:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2687.974746] LustreError: 86502:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.987721] LustreError: 86502:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2692.661103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.089640] LustreError: 86507:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.109644] LustreError: 86507:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2693.172580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.212433] LustreError: 86503:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.247131] LustreError: 86503:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2700.250396] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2703.347133] LustreError: 86506:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.516437] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2709.528969] Lustre: MGS: Client 35b0f4cb-ff78-442a-81c1-a3bf6398ee76 (at 0@lo) reconnecting [ 2709.554281] LustreError: 86517:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2709.843716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.925969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2714.116265] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2730.472440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.473864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2730.474996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2730.482286] Lustre: Skipped 1 previous similar message [ 2730.493612] Lustre: Skipped 1 previous similar message [ 2734.132051] Lustre: server umount lustre-OST0000 complete [ 2740.708763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.709856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2740.710332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.734347] Lustre: Skipped 1 previous similar message [ 2745.905226] Lustre: server umount lustre-MDT0000 complete [ 2745.915146] LustreError: 86501:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.932042] LustreError: 86501:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 2751.094562] Lustre: server umount lustre-MDT0001 complete [ 2756.279707] Lustre: server umount MGS complete [ 2770.293122] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:22:37 (1787826157) [ 2798.741856] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 2809.384290] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2842.666625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.427880] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2857.363120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.563677] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2857.601653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2857.678390] Lustre: lustre-MDT0001: new disk, initializing [ 2857.766651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2857.786793] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2857.794238] Lustre: 93676:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2857.802920] LustreError: 93676:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2862.336651] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2874.026333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.412272] Lustre: lustre-OST0000: new disk, initializing [ 2874.416417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2874.422616] Lustre: 94443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2874.483250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.553617] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2882.657061] Lustre: 94796:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000052a8c7ef x1874671480905856/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787826286 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2891.340561] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2898.546591] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.566641] Lustre: MGS: Client b31bf7e9-0727-4c51-8ccf-c85c60c1abd5 (at 0@lo) reconnecting [ 2898.624853] Lustre: 92946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2898.647631] LustreError: 93662:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2898.682693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2898.792558] Lustre: lustre-MDT0000: new disk, initializing [ 2898.993834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.020990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2899.041778] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2900.066516] Lustre: 95513:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000942c718d x1874671480912768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787826304 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2904.361196] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2905.327263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2905.333646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2905.367343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2909.168485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2909.179153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2935.778642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2935.801988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.842866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2935.847288] Lustre: Skipped 1 previous similar message [ 2941.381157] Lustre: server umount lustre-OST0000 complete [ 2946.542825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.554913] LustreError: Skipped 1 previous similar message [ 2946.560956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.583159] Lustre: Skipped 1 previous similar message [ 2946.590933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.599318] Lustre: Skipped 3 previous similar messages [ 2951.489303] Lustre: server umount lustre-MDT0000 complete [ 2951.653295] LustreError: 95945:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.669226] LustreError: 95945:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2961.749325] Lustre: server umount lustre-MDT0001 complete [ 2965.624973] Lustre: server umount MGS complete [ 2977.565837] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:26:05 (1787826365) [ 2998.040590] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3000.815464] Key type lgssc unregistered [ 3001.085235] LNet: 98016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.092617] LNetError: 98016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3001.111238] LNet: Removed LNI 192.168.201.125@tcp [ 3001.912219] Key type .llcrypt unregistered [ 3001.914449] Key type ._llcrypt unregistered [ 3020.326658] Key type ._llcrypt registered [ 3020.328352] Key type .llcrypt registered [ 3020.657078] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3022.218458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3022.355446] alg: No test for adler32 (adler32-zlib) [ 3023.365889] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3023.564521] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3025.240194] Key type lgssc registered [ 3026.962086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.166618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.189980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.626846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.921793] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3045.860734] LustreError: 99242:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.871112] LustreError: 99242:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3050.591623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.135806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.494366] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3065.703686] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3070.721751] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3077.546557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3077.843490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.949776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3082.990814] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3089.946369] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3106.785767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.801907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.804252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.814559] Lustre: Skipped 1 previous similar message [ 3106.865781] Lustre: Skipped 1 previous similar message [ 3111.920810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.937081] Lustre: Skipped 1 previous similar message [ 3117.024132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3117.028583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.034741] Lustre: Skipped 1 previous similar message [ 3117.141101] Lustre: server umount lustre-OST0000 complete [ 3122.146760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.161853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.167269] Lustre: Skipped 1 previous similar message [ 3127.270887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.282285] Lustre: Skipped 1 previous similar message [ 3128.264236] Lustre: server umount lustre-MDT0000 complete [ 3131.393363] LustreError: 99223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826519 with bad export cookie 15287373490666118709 [ 3131.400775] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.404930] LustreError: 99223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3131.609316] Lustre: server umount lustre-MDT0001 complete [ 3143.494760] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3146.467686] Key type lgssc unregistered [ 3146.773710] LNet: 102800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3146.777897] LNetError: 102800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3146.799480] LNet: Removed LNI 192.168.201.125@tcp [ 3147.561094] Key type .llcrypt unregistered [ 3147.566780] Key type ._llcrypt unregistered [ 3165.387568] Key type ._llcrypt registered [ 3165.389647] Key type .llcrypt registered [ 3165.492981] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3166.956400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3167.010921] alg: No test for adler32 (adler32-zlib) [ 3168.023742] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3168.323556] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3170.088183] Key type lgssc registered [ 3171.759655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3178.860595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3178.869489] LustreError: 103883:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3178.883120] LustreError: 103884:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3178.917313] LustreError: 103883:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3178.929599] LustreError: 103883:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3178.952326] LustreError: 103883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3178.976247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.527516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.753111] LustreError: 103922:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.783500] LustreError: 103922:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3192.255441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.428437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.520414] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3205.299503] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing lsmod [ 3208.473474] LustreError: 105081:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3208.540793] LustreError: 105091:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3208.548540] LustreError: 105081:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4933 [ 3208.557887] LustreError: 105081:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3208.563217] LustreError: 105081:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3208.576035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.062816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.200147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.212194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.212751] Lustre: Skipped 1 previous similar message [ 3212.224879] Lustre: Skipped 2 previous similar messages [ 3217.323828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.339202] Lustre: Skipped 2 previous similar messages [ 3218.758281] Lustre: server umount lustre-MDT0000 complete [ 3222.508522] LustreError: 103922:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.527082] LustreError: 103922:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3223.014193] LustreError: 103900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826611 with bad export cookie 16475530554382619193 [ 3223.032756] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.045962] Lustre: 103914:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.125@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3223.544346] LustreError: 104353:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002d2d35ef x1874672018804736/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3223.865991] Lustre: server umount lustre-MDT0001 complete [ 3238.796558] Lustre: server umount lustre-OST0000 complete [ 3248.196530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.543076] LustreError: 106155:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.608357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.733256] LustreError: 106156:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.108903] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3257.834096] LustreError: 106155:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3266.232487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.633496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3272.869407] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3281.765160] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3288.754094] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3296.839893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.607134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.579341] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3314.629673] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3318.064449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3326.434284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3326.447474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.462258] Lustre: Skipped 1 previous similar message [ 3326.467042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.484116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.484591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.707617] Lustre: server umount lustre-OST0000 complete [ 3338.210856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3338.225362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.242456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.250439] Lustre: Skipped 1 previous similar message [ 3342.260467] Lustre: server umount lustre-MDT0000 complete [ 3343.842037] LustreError: 106156:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.852534] LustreError: 106156:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3346.521965] LustreError: 106137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826734 with bad export cookie 16475530554382619886 [ 3346.538408] LustreError: 106137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3346.552855] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3346.948313] Lustre: server umount lustre-MDT0001 complete [ 3359.084624] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3362.162880] Key type lgssc unregistered [ 3362.472340] LNet: 109701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3362.479901] LNetError: 109701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3362.499887] LNet: Removed LNI 192.168.201.125@tcp [ 3363.354542] Key type .llcrypt unregistered [ 3363.357651] Key type ._llcrypt unregistered [ 3393.193489] Key type ._llcrypt registered [ 3393.196787] Key type .llcrypt registered [ 3393.318957] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3394.278483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3394.448516] alg: No test for adler32 (adler32-zlib) [ 3395.767602] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3396.171133] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3397.936218] Key type lgssc registered [ 3399.528946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3411.511572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3411.532746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.877323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.772459] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3423.215480] LustreError: 111122:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.232286] LustreError: 111122:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3428.340058] LustreError: 111123:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.874405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.453300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3438.018496] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3446.740263] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3453.466611] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3461.302618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3461.850341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.775937] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3479.710453] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3483.460754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3491.146089] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3494.513371] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3498.980717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3498.997576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3499.014103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3499.024534] Lustre: Skipped 1 previous similar message [ 3504.098360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.102635] Lustre: Skipped 1 previous similar message [ 3504.864535] Lustre: server umount lustre-OST0000 complete [ 3509.226407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.230375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3509.243834] Lustre: Skipped 2 previous similar messages [ 3514.339492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.356899] Lustre: Skipped 2 previous similar messages [ 3515.823716] Lustre: server umount lustre-MDT0000 complete [ 3519.458465] LustreError: 111118:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.480449] LustreError: 111118:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3520.215784] LustreError: 111857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826908 with bad export cookie 7838150453431740126 [ 3520.228979] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.245234] LustreError: 111857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3520.791758] Lustre: server umount lustre-MDT0001 complete [ 3534.007039] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3537.529974] Key type lgssc unregistered [ 3537.924226] LNet: 114776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3537.932459] LNetError: 114776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3537.945770] LNet: Removed LNI 192.168.201.125@tcp [ 3539.201493] Key type .llcrypt unregistered [ 3539.214616] Key type ._llcrypt unregistered [ 3565.810410] Key type ._llcrypt registered [ 3565.815306] Key type .llcrypt registered [ 3565.998747] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3567.455651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3567.597821] alg: No test for adler32 (adler32-zlib) [ 3568.790812] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3569.114828] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3570.760491] Key type lgssc registered [ 3572.256323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3580.481926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3580.508344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.909246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.742672] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3592.172677] LustreError: 116002:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3592.200901] LustreError: 116002:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3595.336531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.074215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.640440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3610.519821] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3616.967070] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3623.199901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3623.422846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.524333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3630.373440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3639.314303] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3651.041844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3651.047928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.071102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.079400] Lustre: Skipped 1 previous similar message [ 3651.097196] Lustre: Skipped 1 previous similar message [ 3655.583887] Lustre: server umount lustre-OST0000 complete [ 3661.284361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.296778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.302226] Lustre: Skipped 1 previous similar message [ 3661.317144] Lustre: Skipped 1 previous similar message [ 3666.401633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.421534] Lustre: Skipped 1 previous similar message [ 3671.531346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3671.534949] Lustre: Skipped 1 previous similar message [ 3674.080335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3674.388890] Lustre: server umount lustre-MDT0000 complete [ 3676.664418] LustreError: 115999:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.687162] LustreError: 115999:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3678.879348] LustreError: 115982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827067 with bad export cookie 12802646112494621819 [ 3678.895329] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3678.900426] LustreError: 115982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3679.680615] Lustre: server umount lustre-MDT0001 complete [ 3693.348702] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3697.052953] Key type lgssc unregistered [ 3697.542476] LNet: 119540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3697.558328] LNetError: 119540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3697.584714] LNet: Removed LNI 192.168.201.125@tcp [ 3698.651522] Key type .llcrypt unregistered [ 3698.653587] Key type ._llcrypt unregistered [ 3729.547135] Key type ._llcrypt registered [ 3729.550775] Key type .llcrypt registered [ 3729.742973] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3731.635342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3731.678281] alg: No test for adler32 (adler32-zlib) [ 3732.926707] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3733.366412] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3735.144541] Key type lgssc registered [ 3736.808787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3747.819880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3747.851423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.344601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.093957] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3759.595876] LustreError: 120959:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.619393] LustreError: 120959:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3763.202611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.526649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.787530] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3778.232890] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3786.420647] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.606125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.998438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3804.859927] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3814.933794] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3819.680137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3827.666612] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3832.121780] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3832.127174] Lustre: Skipped 2 previous similar messages [ 3833.910696] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3833.936794] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3836.986229] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3836.992762] Lustre: Skipped 2 previous similar messages [ 3839.281698] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3839.290839] Lustre: Skipped 1 previous similar message [ 3844.736318] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.25@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3844.752561] Lustre: Skipped 2 previous similar messages [ 3849.341487] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.25@tcp 192.168.0.[2,10]@tcp [ 3853.596755] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.25/24@tcp in log lustre-MDT0000 [ 3853.614207] Lustre: Skipped 2 previous similar messages [ 3858.598313] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.25/24@tcp [ 3858.613427] Lustre: Skipped 1 previous similar message [ 3865.063583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3865.068590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3865.077101] LustreError: Skipped 1 previous similar message [ 3865.077857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3865.117703] Lustre: Skipped 1 previous similar message [ 3866.085222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3866.097344] Lustre: Skipped 2 previous similar messages [ 3870.623416] Lustre: server umount lustre-OST0000 complete [ 3876.326377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.358249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3876.365673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3876.379524] LustreError: Skipped 4 previous similar messages [ 3876.404697] Lustre: Skipped 1 previous similar message [ 3881.368669] Lustre: server umount lustre-MDT0000 complete [ 3881.442963] LustreError: 120960:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3881.463221] LustreError: 120960:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3885.890229] LustreError: 120940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827274 with bad export cookie 5556412048840763361 [ 3885.906469] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3886.368799] Lustre: server umount lustre-MDT0001 complete [ 3900.148275] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3903.361961] Key type lgssc unregistered [ 3903.712394] LNet: 125768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3903.724660] LNetError: 125768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3903.741345] LNet: Removed LNI 192.168.201.125@tcp [ 3904.754720] Key type .llcrypt unregistered [ 3904.757474] Key type ._llcrypt unregistered [ 3939.106629] Key type ._llcrypt registered [ 3939.111170] Key type .llcrypt registered [ 3939.251394] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3941.330969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3941.382272] alg: No test for adler32 (adler32-zlib) [ 3942.596897] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3943.007723] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3944.760245] Key type lgssc registered [ 3946.324714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3956.531643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3956.543186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.059786] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3958.110324] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.25/24@tcp [ 3958.120484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.657520] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3968.492762] LustreError: 127330:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.516045] LustreError: 127330:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3973.608099] LustreError: 127331:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3977.515270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.180320] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3978.227899] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.25/24@tcp [ 3978.294669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3984.949662] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3994.700883] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4002.677628] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4011.842791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.503957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.712934] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4029.051544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4033.086561] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4049.384606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.388734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.393019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.405858] Lustre: Skipped 1 previous similar message [ 4051.099077] Lustre: server umount lustre-OST0000 complete [ 4055.017656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.026728] LustreError: Skipped 1 previous similar message [ 4055.034268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.071790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4055.084801] Lustre: Skipped 1 previous similar message [ 4059.618721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.624961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4059.674344] Lustre: Skipped 1 previous similar message [ 4061.366905] Lustre: server umount lustre-MDT0000 complete [ 4064.741667] LustreError: 127326:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.763386] LustreError: 127326:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4065.542402] LustreError: 127312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827453 with bad export cookie 15991845371496931160 [ 4065.552449] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.556497] LustreError: 127312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4066.033606] Lustre: server umount lustre-MDT0001 complete [ 4079.016508] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4082.759311] Key type lgssc unregistered [ 4083.124596] LNet: 130926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4083.142116] LNetError: 130926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4083.160990] LNet: Removed LNI 192.168.201.125@tcp [ 4084.063257] Key type .llcrypt unregistered [ 4084.065478] Key type ._llcrypt unregistered [ 4113.582977] Key type ._llcrypt registered [ 4113.584843] Key type .llcrypt registered [ 4113.680473] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 4132.874801] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4134.989423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4135.237782] alg: No test for adler32 (adler32-zlib) [ 4136.527269] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 4136.806587] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4138.544195] Key type lgssc registered [ 4140.751747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4190.291886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4190.327410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.764515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4191.795989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4191.891699] Lustre: lustre-MDT0000: new disk, initializing [ 4191.976858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.998854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4197.724815] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4210.277287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.447962] Lustre: 134336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4210.598934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4210.608019] Lustre: Skipped 1 previous similar message [ 4210.805541] Lustre: lustre-MDT0001: new disk, initializing [ 4211.101088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4211.166151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4211.176949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4216.749248] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4227.531889] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4232.866802] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4240.480660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.801966] Lustre: lustre-OST0000: new disk, initializing [ 4240.805707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4240.815545] Lustre: 136304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4240.901808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.297510] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4248.747044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4248.790651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4248.956870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4259.030494] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4264.966786] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4265.282798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4271.237988] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4271.675590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.660741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.665614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.689150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.706373] Lustre: Skipped 1 previous similar message [ 4279.796597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.812601] Lustre: Skipped 1 previous similar message [ 4280.909531] Lustre: server umount lustre-OST0000 complete [ 4287.980512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4287.999525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.015743] Lustre: Skipped 1 previous similar message [ 4288.022642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.020322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.047512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.059641] Lustre: Skipped 1 previous similar message [ 4295.138618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4295.144579] Lustre: Skipped 1 previous similar message [ 4300.256182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4300.600322] Lustre: server umount lustre-MDT0000 complete [ 4305.121716] LustreError: 134326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827693 with bad export cookie 1724281127307039440 [ 4305.134390] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4305.143353] LustreError: 134326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4305.497744] Lustre: server umount lustre-MDT0001 complete [ 4314.296302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.846342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.032335] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4320.233128] LustreError: 138522:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.277485] LustreError: 138522:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4325.346205] LustreError: 138521:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.057318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.659068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4335.297695] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4342.667932] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4348.821435] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4355.612912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.123749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.518080] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4370.651952] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4376.506516] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4376.740360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4382.299530] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4382.583635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4393.919269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4394.145768] Lustre: lustre-OST0001: new disk, initializing [ 4394.148764] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4394.154234] Lustre: 141894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4394.256117] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4401.027178] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4403.303070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4403.310895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4403.417464] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4409.991992] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4410.197560] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4415.592983] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4415.835342] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4421.735249] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bef18f4e000.ost_server_uuid 50 [ 4423.355909] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bef18f4e000.ost_server_uuid in IDLE state after 0 sec [ 4434.401655] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4434.412931] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.432764] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4434.442491] Lustre: Skipped 3 previous similar messages [ 4446.176439] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4446.266798] Lustre: server umount lustre-OST0001 complete [ 4449.767162] LustreError: 142249:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.776869] LustreError: 142249:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4451.810216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4451.818678] LustreError: Skipped 1 previous similar message [ 4451.824684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.847078] Lustre: Skipped 1 previous similar message [ 4451.854840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.860180] Lustre: Skipped 4 previous similar messages [ 4454.893138] LustreError: 140341:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.915463] LustreError: 140341:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4456.919791] Lustre: server umount lustre-OST0000 complete [ 4462.562455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.578723] LustreError: Skipped 1 previous similar message [ 4462.582564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.604444] Lustre: Skipped 1 previous similar message [ 4475.360147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4475.738497] Lustre: server umount lustre-MDT0000 complete [ 4479.344666] LustreError: 138504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827867 with bad export cookie 1724281127307040322 [ 4479.350325] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.360657] LustreError: 138504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4479.774893] Lustre: server umount lustre-MDT0001 complete [ 4501.511947] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4505.771281] Key type lgssc unregistered [ 4506.127241] LNet: 144907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4506.143568] LNetError: 144907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4506.173279] LNet: Removed LNI 192.168.201.125@tcp [ 4507.130385] Key type .llcrypt unregistered [ 4507.133678] Key type ._llcrypt unregistered [ 4559.410676] Key type ._llcrypt registered [ 4559.415271] Key type .llcrypt registered [ 4559.561475] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 4577.526828] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4578.812890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4578.985478] alg: No test for adler32 (adler32-zlib) [ 4580.094753] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 4580.418440] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4582.096623] Key type lgssc registered [ 4583.233736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4616.941720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4616.952222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.216746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4618.244557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4618.298511] Lustre: lustre-MDT0000: new disk, initializing [ 4618.353118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4618.365670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4622.468882] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4632.553660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.705480] Lustre: 149064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4632.776309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4632.783499] Lustre: Skipped 1 previous similar message [ 4632.918581] Lustre: lustre-MDT0001: new disk, initializing [ 4633.079709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4633.125811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4633.141262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4637.917678] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4648.589028] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4655.284742] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.384741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.624484] Lustre: lustre-OST0000: new disk, initializing [ 4662.626778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4662.631676] Lustre: 151031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4662.697413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.883500] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4670.585609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4670.604603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4670.718319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4677.930587] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4682.460893] Lustre: Failing over lustre-OST0000 [ 4682.594380] Lustre: server umount lustre-OST0000 complete [ 4699.616148] Lustre: 147290:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828071/real 1787828071] req@00000000c9649e9c x1874673499407360/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787828087 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4699.668074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.671951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.212846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.247523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.266522] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4702.673289] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828074/real 1787828074] req@0000000030d49638 x1874673499408640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787828090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4704.349946] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4706.784482] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828079/real 1787828079] req@00000000a43b101e x1874673499409792/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787828095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4706.865674] Lustre: 147289:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4710.339732] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4710.385235] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4710.386580] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4711.904173] Lustre: 147288:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828084/real 1787828084] req@00000000abbb5531 x1874673499410816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787828100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4711.920921] Lustre: 147288:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4713.858226] Lustre: Failing over lustre-MDT0000 [ 4714.567800] Lustre: server umount lustre-MDT0000 complete [ 4714.977221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4714.982451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.990552] Lustre: Skipped 2 previous similar messages [ 4720.124531] LustreError: 149076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.156501] LustreError: 149076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 4725.217496] LustreError: 149077:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.246866] LustreError: 149077:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4730.345399] LustreError: 149076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4730.386365] LustreError: 149076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4731.360128] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828103/real 1787828103] req@00000000e11fa824 x1874673499420928/t0(0) o400->MGC192.168.201.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787828119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4731.387456] Lustre: 147291:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4731.397529] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.407658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.947376] LustreError: 149073:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.973405] LustreError: 149073:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 4741.042850] LustreError: 149072:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.063787] LustreError: 149072:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 4741.608607] LustreError: 147287:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009d9cb330 x1874673499426432/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4741.917369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.971330] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4742.077937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4746.508072] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4747.245919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4747.252911] Lustre: Skipped 1 previous similar message [ 4747.285983] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4747.335259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4756.449653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4756.461668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.481610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.475307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4762.596528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4762.604026] Lustre: Skipped 2 previous similar messages [ 4767.720868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4768.224617] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4768.501863] Lustre: server umount lustre-OST0000 complete [ 4772.834679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.838682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.854102] Lustre: Skipped 2 previous similar messages [ 4772.854414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4772.881968] Lustre: Skipped 2 previous similar messages [ 4778.752618] Lustre: server umount lustre-MDT0000 complete [ 4782.312606] LustreError: 149056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828170 with bad export cookie 7011874167025768088 [ 4782.327780] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.328838] LustreError: 149056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4782.793794] Lustre: server umount lustre-MDT0001 complete [ 4794.989712] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4797.415392] Key type lgssc unregistered [ 4797.745573] LNet: 154703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4797.751490] LNetError: 154703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4797.782553] LNet: Removed LNI 192.168.201.125@tcp [ 4798.520269] Key type .llcrypt unregistered [ 4798.522820] Key type ._llcrypt unregistered [ 4839.000758] Key type ._llcrypt registered [ 4839.002270] Key type .llcrypt registered [ 4839.165832] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4840.467491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4840.694581] alg: No test for adler32 (adler32-zlib) [ 4841.903427] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 4842.248289] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4843.912305] Key type lgssc registered [ 4845.199583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4853.901116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4853.923172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.595834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.561768] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4864.998274] LustreError: 156224:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.014185] LustreError: 156224:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4870.118449] LustreError: 156225:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.306596] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4875.235221] LustreError: 156224:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.107235] LustreError: 156253:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4877.146848] LustreError: 156253:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4877.160564] LustreError: 156253:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4877.787660] Lustre: server umount lustre-MDT0000 complete [ 4893.835248] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 4904.649781] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4941.528736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.775609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4941.791594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4941.844371] Lustre: lustre-MDT0000: new disk, initializing [ 4941.884467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.895512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4946.954375] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4959.730249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.860887] Lustre: 159554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.925228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4959.929352] Lustre: Skipped 1 previous similar message [ 4960.056385] Lustre: lustre-MDT0001: new disk, initializing [ 4960.203437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4960.254953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4960.265810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4965.856953] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4976.136798] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4982.676295] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4991.464706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4991.831417] Lustre: lustre-OST0000: new disk, initializing [ 4991.834276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4991.842909] Lustre: 161525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4991.937770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4997.208314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4997.239405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4997.317148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4998.038807] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5009.123330] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5012.369860] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5015.131244] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:00:02 (1787828402) [ 5262.305966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5262.325305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.355046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5263.842654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.843429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5263.867222] Lustre: Skipped 1 previous similar message [ 5263.879533] Lustre: Skipped 2 previous similar messages [ 5267.078756] Lustre: server umount lustre-MDT0000 complete [ 5268.966326] LustreError: 160720:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.983472] LustreError: 160720:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5271.965958] LustreError: 159549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828660 with bad export cookie 14033260615520516664 [ 5271.978906] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5274.081771] LustreError: 162197:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.081771] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.125094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5276.128926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5281.249342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5282.273041] LustreError: 160719:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.295545] LustreError: 160719:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5286.368455] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5286.584241] Lustre: server umount lustre-MDT0001 complete [ 5305.314641] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5305.471086] Lustre: server umount lustre-OST0000 complete [ 5315.252758] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5325.499406] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5366.976149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.239752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5367.292143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5367.439936] Lustre: lustre-MDT0000: new disk, initializing [ 5367.550136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.595394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5373.845807] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5385.452979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.593437] Lustre: 166205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5385.642651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5385.647187] Lustre: Skipped 1 previous similar message [ 5385.749680] Lustre: lustre-MDT0001: new disk, initializing [ 5385.848221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.885215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5385.897343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5391.054387] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5401.497739] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5407.728108] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5416.231500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5416.670752] Lustre: lustre-OST0000: new disk, initializing [ 5416.678612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5416.690842] Lustre: 168175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5416.825342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.658027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5422.672320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5422.812490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5424.802321] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5436.635778] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5442.521570] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5442.748960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5448.109410] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5448.366731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5453.798236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.802728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.806715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.806724] Lustre: Skipped 1 previous similar message [ 5453.848509] Lustre: Skipped 1 previous similar message [ 5456.766250] Lustre: server umount lustre-OST0000 complete [ 5462.503413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5462.518980] LustreError: Skipped 1 previous similar message [ 5462.526476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.581548] Lustre: server umount lustre-MDT0000 complete [ 5469.154027] LustreError: 166216:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5469.185923] LustreError: 166216:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5471.289531] LustreError: 166197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828859 with bad export cookie 14033260615520675487 [ 5471.305459] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.308287] LustreError: 166197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5471.598277] Lustre: server umount lustre-MDT0001 complete [ 5480.843601] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:07:48 (1787828868) [ 5504.168460] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5516.125411] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5561.342943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.546253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5561.574434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5561.679716] Lustre: lustre-MDT0000: new disk, initializing [ 5561.765454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.781850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5566.375963] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5578.625658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.740510] Lustre: 172935:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5579.269203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5585.436907] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5596.558210] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5603.430724] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5611.239866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5611.565370] Lustre: lustre-OST0000: new disk, initializing [ 5611.568508] Lustre: Skipped 1 previous similar message [ 5611.571736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5611.575584] Lustre: Skipped 2 previous similar messages [ 5611.585533] Lustre: 174911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5617.737851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5617.752913] Lustre: Skipped 1 previous similar message [ 5617.758979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5617.855493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5619.929369] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5630.329260] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5638.637300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5638.650508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.675401] Lustre: Skipped 1 previous similar message [ 5638.680105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5638.687968] Lustre: Skipped 5 previous similar messages [ 5644.356926] Lustre: server umount lustre-OST0000 complete [ 5655.156270] Lustre: server umount lustre-MDT0000 complete [ 5659.106513] LustreError: 175492:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5659.141706] LustreError: 175492:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5659.880918] LustreError: 174910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829048 with bad export cookie 14033260615520676376 [ 5659.888462] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5659.896927] LustreError: 174910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5660.163682] Lustre: server umount lustre-MDT0001 complete [ 5669.826456] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:10:57 (1787829057) [ 5688.866314] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5697.497900] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5739.644270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.857393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.882104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5739.945787] Lustre: lustre-MDT0000: new disk, initializing [ 5740.052528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5740.056318] Lustre: Skipped 2 previous similar messages [ 5740.068585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5745.686646] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5755.948748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.061472] Lustre: 179242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5756.286832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5761.003914] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5770.269438] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5775.756301] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5782.735743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5783.000333] Lustre: 181219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5789.622215] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5792.383818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5792.622786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5799.485645] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5813.221188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5813.232841] LustreError: Skipped 1 previous similar message [ 5813.243369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.246483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.265843] Lustre: Skipped 4 previous similar messages [ 5813.287705] Lustre: Skipped 7 previous similar messages [ 5814.702403] Lustre: server umount lustre-OST0000 complete [ 5822.949808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5822.952835] LustreError: Skipped 1 previous similar message [ 5825.782146] Lustre: server umount lustre-MDT0000 complete [ 5828.580584] LustreError: 179254:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.616516] LustreError: 179254:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5830.304114] LustreError: 179234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829218 with bad export cookie 14033260615520677454 [ 5830.310533] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.339506] LustreError: 179234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5831.101324] Lustre: server umount lustre-MDT0001 complete [ 5845.738440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5848.913346] Key type lgssc unregistered [ 5849.269143] LNet: 183078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5849.278489] LNetError: 183078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5849.296301] LNet: Removed LNI 192.168.201.125@tcp [ 5850.842323] Key type .llcrypt unregistered [ 5850.845225] Key type ._llcrypt unregistered [ 5880.875262] Key type ._llcrypt registered [ 5880.876857] Key type .llcrypt registered [ 5881.062377] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5882.259569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.381442] alg: No test for adler32 (adler32-zlib) [ 5883.705429] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 5883.953208] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5885.672294] Key type lgssc registered [ 5886.813174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5895.332519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5895.345818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.994157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.681490] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5907.427679] LustreError: 184495:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.452563] LustreError: 184495:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5912.192163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.643487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5917.789882] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5925.817901] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5932.011802] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5939.106802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.375331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.505617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5946.387827] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5954.891697] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5978.594857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.597949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.630832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.636902] Lustre: Skipped 1 previous similar message [ 5983.062881] Lustre: server umount lustre-OST0000 complete [ 5988.833910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.848991] Lustre: Skipped 1 previous similar message [ 5988.860749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5988.869762] Lustre: Skipped 1 previous similar message [ 5993.211165] Lustre: server umount lustre-MDT0000 complete [ 5993.959307] LustreError: 185464:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.978992] LustreError: 185464:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5997.220525] LustreError: 184474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829385 with bad export cookie 3115026015463715890 [ 5997.223449] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.224665] LustreError: 184474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5997.639375] Lustre: server umount lustre-MDT0001 complete [ 6010.784582] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 6014.105177] Key type lgssc unregistered [ 6014.443144] LNet: 188059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6014.447721] LNetError: 188059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6014.472278] LNet: Removed LNI 192.168.201.125@tcp [ 6015.470180] Key type .llcrypt unregistered [ 6015.472185] Key type ._llcrypt unregistered [ 6042.006688] Key type ._llcrypt registered [ 6042.011829] Key type .llcrypt registered [ 6042.165225] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6043.780277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6043.895418] alg: No test for adler32 (adler32-zlib) [ 6045.137058] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 6045.473563] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 6047.152665] Key type lgssc registered [ 6048.414184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.514402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6057.548348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.985702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.689802] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6069.231361] LustreError: 189477:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.253966] LustreError: 189477:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6072.616148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6072.890962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6078.122578] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6086.434410] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6092.834677] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6099.342272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6099.737898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.870426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6105.865991] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6112.826831] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6123.492357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.494260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.519585] Lustre: Skipped 1 previous similar message [ 6123.542034] Lustre: Skipped 1 previous similar message [ 6125.114714] Lustre: server umount lustre-OST0000 complete [ 6131.927451] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9bef0466e800.ost_server_uuid 50 [ 6145.251180] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6162.787212] Lustre: server umount lustre-MDT0000 complete [ 6163.300992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.353874] LustreError: 190205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.393548] LustreError: 190205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6167.475299] LustreError: 189458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829555 with bad export cookie 11859520114357858820 [ 6167.485334] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6167.487098] LustreError: 189458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6167.824563] Lustre: server umount lustre-MDT0001 complete [ 6177.388060] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:19:24 (1787829564) [ 6184.452041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.922364] LustreError: 193414:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.988616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.957711] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6190.061337] LustreError: 193415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.171568] LustreError: 193414:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.649632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.239345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6204.706166] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6214.103422] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6220.594653] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6227.775639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6228.163763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.277629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6235.011440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6244.260456] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6252.878687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6253.160742] Lustre: lustre-OST0001: new disk, initializing [ 6253.171501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6253.183364] Lustre: 196255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6253.348148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.243679] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6263.369220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6263.382623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6263.495920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6271.126109] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6274.026114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.055598] Lustre: Skipped 2 previous similar messages [ 6274.067461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6274.080488] Lustre: Skipped 1 previous similar message [ 6277.074768] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 6279.143435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.174610] Lustre: Skipped 1 previous similar message [ 6279.739924] Lustre: server umount lustre-OST0000 complete [ 6282.178988] LustreError: 196717:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.471454] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6287.710492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6292.857496] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6293.073971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6299.624802] LustreError: 195239:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.647894] LustreError: 195239:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 6309.869413] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6309.871713] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.895023] LustreError: Skipped 1 previous similar message [ 6309.906713] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6314.645413] Lustre: server umount lustre-OST0001 complete [ 6320.102425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.112737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6320.120302] Lustre: Skipped 2 previous similar messages [ 6320.135992] Lustre: Skipped 2 previous similar messages [ 6324.794427] Lustre: server umount lustre-MDT0000 complete [ 6328.706136] LustreError: 193394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829716 with bad export cookie 11859520114357859891 [ 6328.709883] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6328.721754] LustreError: 193394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6329.185523] Lustre: server umount lustre-MDT0001 complete [ 6355.559796] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:22:22 (1787829742) [ 6362.282640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.653575] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6362.686926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6362.839264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.227372] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6378.008250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.113312] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6378.149516] Lustre: 199696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6383.441376] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6392.513337] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6398.464725] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.198345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.415816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6413.004012] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6413.277290] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6420.089441] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6428.513405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6428.570084] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6428.791385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6428.801080] Lustre: Skipped 2 previous similar messages [ 6436.786928] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6445.143548] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.610191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.617352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.639143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6454.416545] Lustre: server umount lustre-OST0000 complete [ 6456.262681] LustreError: 202885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6456.298237] LustreError: 202885:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 6472.171960] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6472.177697] LustreError: Skipped 1 previous similar message [ 6472.181196] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.189632] Lustre: Skipped 1 previous similar message [ 6477.254241] Lustre: server umount lustre-OST0001 complete [ 6482.402204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6482.408085] Lustre: Skipped 7 previous similar messages [ 6495.200216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6495.390454] Lustre: server umount lustre-MDT0000 complete [ 6498.985641] LustreError: 199689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829887 with bad export cookie 11859520114357861298 [ 6498.998903] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.999454] LustreError: 199689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6499.559288] Lustre: server umount lustre-MDT0001 complete [ 6523.075797] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:25:10 (1787829910) [ 6539.599783] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 6550.250531] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 6590.186050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.573534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6590.618712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6590.777364] Lustre: lustre-MDT0000: new disk, initializing [ 6590.884709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6590.932987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6597.137703] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6611.614379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.738355] Lustre: 207794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6611.800711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6611.803720] Lustre: Skipped 1 previous similar message [ 6611.874869] Lustre: lustre-MDT0001: new disk, initializing [ 6611.994142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6612.014947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6616.938719] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6627.785468] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6636.013449] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6644.426636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6644.703754] Lustre: lustre-OST0000: new disk, initializing [ 6644.712322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6644.717222] Lustre: 209768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6652.312167] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6653.021390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6653.037536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6653.236195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6662.726881] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6667.672490] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6667.947839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6673.007688] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6673.331852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6679.009182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6679.011798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.024670] LustreError: Skipped 2 previous similar messages [ 6679.028791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6679.028801] Lustre: Skipped 4 previous similar messages [ 6679.072434] Lustre: Skipped 4 previous similar messages [ 6681.967543] Lustre: server umount lustre-OST0000 complete [ 6688.738363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6692.191972] Lustre: server umount lustre-MDT0000 complete [ 6694.388857] LustreError: 207806:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.418210] LustreError: 207806:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 13 previous similar messages [ 6696.706666] LustreError: 209767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830084 with bad export cookie 11859520114357862859 [ 6696.711112] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6696.727966] LustreError: 209767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6697.223683] Lustre: server umount lustre-MDT0001 complete [ 6706.810769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.793268] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6722.264368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.862834] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6735.724501] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6742.254801] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6748.645718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6757.621461] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6766.162799] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6772.986921] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6773.253129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6779.025540] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6779.228228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6782.945815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.968095] Lustre: Skipped 3 previous similar messages [ 6782.992411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6787.620854] Lustre: server umount lustre-OST0000 complete [ 6796.174804] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6802.657458] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6831.780926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6839.323622] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6847.955736] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6848.576157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6848.593893] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6848.594610] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6850.700537] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6858.724425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6858.728857] Lustre: Skipped 6 previous similar messages [ 6861.774189] Lustre: server umount lustre-OST0000 complete [ 6866.401897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6880.224197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6880.590212] Lustre: server umount lustre-MDT0000 complete [ 6883.615534] LustreError: 211965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830271 with bad export cookie 11859520114357863741 [ 6883.618187] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6883.623337] LustreError: 211965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6883.998803] Lustre: server umount lustre-MDT0001 complete [ 6891.677282] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:31:19 (1787830279) [ 6898.073108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.468724] LustreError: 217956:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.480940] LustreError: 217956:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 6898.515154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6898.518977] Lustre: Skipped 6 previous similar messages [ 6903.047720] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6911.377410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.510838] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6924.796624] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6931.233116] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6938.702585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.225177] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6954.571799] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6960.112062] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6960.293800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6966.065624] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6966.431423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6974.759778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6975.065874] Lustre: lustre-OST0001: new disk, initializing [ 6975.071547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6975.080868] Lustre: 221328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6982.295377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6982.309113] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6982.441087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6982.931840] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 6994.033416] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6999.607805] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6999.774456] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7004.739702] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7004.967207] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7008.231705] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.237617] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.244947] LustreError: Skipped 1 previous similar message [ 7008.270497] Lustre: Skipped 5 previous similar messages [ 7013.131274] Lustre: server umount lustre-OST0001 complete [ 7021.431722] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7021.635796] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7027.042815] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7027.374763] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7058.909942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.515956] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7064.554057] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7064.587553] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7064.588493] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7064.630210] Lustre: Skipped 1 previous similar message [ 7072.195954] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7073.878684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7079.906027] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7079.940417] LustreError: Skipped 1 previous similar message [ 7082.222310] Lustre: server umount lustre-OST0001 complete [ 7106.939907] Lustre: server umount lustre-OST0000 complete [ 7115.750637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.754766] Lustre: Skipped 18 previous similar messages [ 7117.936694] Lustre: server umount lustre-MDT0000 complete [ 7121.961313] LustreError: 217936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830510 with bad export cookie 11859520114357864700 [ 7121.963934] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.978144] LustreError: 217936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7128.604437] Lustre: server umount lustre-MDT0001 complete [ 7155.903953] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:35:43 (1787830543) [ 7163.047840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.502723] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7163.560261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7169.494393] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7179.002837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.079061] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7179.102661] Lustre: 226570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7184.846724] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7192.886416] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7199.229422] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7206.559298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7206.772853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7213.735051] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7221.125516] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7230.831708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7230.891707] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7237.411045] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7246.834692] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7248.990731] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7254.307520] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7254.640534] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7260.371556] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7260.587200] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7267.799228] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bef08d7a000.ost_server_uuid 50 [ 7270.128409] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bef08d7a000.ost_server_uuid in IDLE state after 0 sec [ 7271.639573] Lustre: Permanently deactivating lustre-OST0001 [ 7271.676247] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7276.571115] Lustre: Permanently reactivating lustre-OST0001 [ 7276.599280] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7276.604233] Lustre: Skipped 2 previous similar messages [ 7284.202553] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7284.218690] LustreError: Skipped 3 previous similar messages [ 7284.230700] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.239373] Lustre: Skipped 6 previous similar messages [ 7288.993242] Lustre: server umount lustre-OST0001 complete [ 7289.324758] LustreError: 228414:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.357509] LustreError: 228414:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 41 previous similar messages [ 7308.257682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7308.667927] Lustre: server umount lustre-OST0000 complete [ 7318.955874] Lustre: server umount lustre-MDT0000 complete [ 7323.507241] LustreError: 226560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830711 with bad export cookie 11859520114357865995 [ 7323.508211] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7323.529654] LustreError: 226560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7324.080801] Lustre: server umount lustre-MDT0001 complete [ 7349.372609] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:38:56 (1787830736) [ 7358.206204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.968178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7359.062221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7359.080542] Lustre: Skipped 2 previous similar messages [ 7365.499897] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7377.012805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.233922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7377.337765] Lustre: 233272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7383.153729] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7391.592318] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7398.101388] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7405.683461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.051750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7406.093663] Lustre: Permanently deactivating lustre-OST0000 [ 7406.103486] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7413.591605] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7416.408678] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7422.206713] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7429.759747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.847970] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7430.036965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7430.039676] Lustre: Skipped 11 previous similar messages [ 7435.951147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7436.828221] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7443.947614] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7445.597391] Lustre: Permanently reactivating lustre-OST0000 [ 7445.621759] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7445.633155] Lustre: Skipped 2 previous similar messages [ 7451.874368] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 000000000c90933c, cur 1787830840 deadline 1787830834 last 1787830804 [ 7451.878188] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7451.935078] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7451.960507] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7452.080694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7476.193206] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7476.458919] Lustre: server umount lustre-OST0001 complete [ 7487.713402] Lustre: server umount lustre-OST0000 complete [ 7498.821942] Lustre: server umount lustre-MDT0000 complete [ 7503.283908] LustreError: 236865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830891 with bad export cookie 11859520114357867668 [ 7503.286818] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7503.289375] LustreError: 236865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7503.577195] Lustre: server umount lustre-MDT0001 complete [ 7516.393453] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7520.063952] Key type lgssc unregistered [ 7520.517854] LNet: 238156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7520.534630] LNetError: 238156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.559707] LNet: Removed LNI 192.168.201.125@tcp [ 7521.542382] Key type .llcrypt unregistered [ 7521.544948] Key type ._llcrypt unregistered [ 7546.744324] Key type ._llcrypt registered [ 7546.746271] Key type .llcrypt registered [ 7546.896627] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7547.962591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7548.082964] alg: No test for adler32 (adler32-zlib) [ 7549.318927] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 7549.645654] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7551.392277] Key type lgssc registered [ 7552.675346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7563.987801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7564.015668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.516423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.345773] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7575.013723] LustreError: 239623:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.031951] LustreError: 239623:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7579.430480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7579.527707] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7579.564547] Lustre: 239613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7579.577468] Lustre: Permanently deactivating lustre-MDT0001 [ 7579.589507] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7579.917907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7582.088821] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7582.099249] LustreError: 239654:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7582.116519] LustreError: 239654:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7582.144272] LustreError: 239654:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7585.255904] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7593.403425] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7600.197458] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7607.496280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7607.968111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.048472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7610.162380] LustreError: 241465:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.404855] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7615.473170] LustreError: 241601:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.586669] LustreError: 241467:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.440667] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7625.719280] LustreError: 241601:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.754073] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7625.754220] LustreError: 241601:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7632.137537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7632.380341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7637.499046] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7637.523907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7638.994646] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7648.750642] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7652.335787] Lustre: Permanently reactivating lustre-MDT0001 [ 7652.358861] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7652.367569] Lustre: Skipped 2 previous similar messages [ 7657.434288] Lustre: lustre-MDT0001: Client 4170b4cd-c8e6-4bee-9b88-f7708eb72b71 (at 192.168.201.25@tcp) reconnecting [ 7665.280799] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7667.037566] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7673.736899] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7673.976600] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7679.218729] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7679.496503] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7682.502495] Lustre: Permanently deactivating lustre-MDT0001 [ 7682.520070] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7682.529541] Lustre: Skipped 2 previous similar messages [ 7692.366641] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7700.961823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.970905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7700.992475] Lustre: Skipped 3 previous similar messages [ 7701.000130] Lustre: Skipped 2 previous similar messages [ 7706.067495] Lustre: server umount lustre-MDT0000 complete [ 7706.098454] LustreError: 239623:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.119675] LustreError: 239623:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 7710.195029] LustreError: 243170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831098 with bad export cookie 6593475667757170987 [ 7710.197608] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7710.211889] LustreError: 243170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7710.714823] Lustre: server umount lustre-MDT0001 complete [ 7716.865855] Lustre: server umount lustre-OST0000 complete [ 7730.868666] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:45:18 (1787831118) [ 7742.170447] Lustre: server umount lustre-OST0001 complete [ 7751.276680] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 7762.745783] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7802.416240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7802.713736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7802.723764] Lustre: Skipped 2 previous similar messages [ 7802.751237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7802.878361] Lustre: lustre-MDT0000: new disk, initializing [ 7803.016288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7803.054636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7808.323317] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7820.895137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.156967] Lustre: 247906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7821.299317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7821.312622] Lustre: Skipped 1 previous similar message [ 7821.451719] Lustre: lustre-MDT0001: new disk, initializing [ 7821.726697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7821.789953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7821.809651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7828.335748] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7839.585300] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7847.337500] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7856.449861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.963282] Lustre: lustre-OST0000: new disk, initializing [ 7856.968111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7856.973720] Lustre: 249877:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7857.106952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7864.859844] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7866.942827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7866.959370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7867.129655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7875.611094] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7877.575091] LustreError: 248954:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7886.631326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.819498] Lustre: lustre-OST0001: new disk, initializing [ 7886.821724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7886.825567] Lustre: 250989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7887.656101] LustreError: 248954:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7887.686070] LustreError: 247912:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7893.201350] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7896.616698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7896.636423] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7896.743105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7897.752125] LustreError: 247912:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7903.071710] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7905.000582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7907.297976] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7907.316644] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.329939] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7907.350991] Lustre: Skipped 1 previous similar message [ 7912.421304] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7912.435066] Lustre: Skipped 2 previous similar messages [ 7913.320967] Lustre: server umount lustre-OST0001 complete [ 7917.552185] LustreError: 249862:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.569494] LustreError: 249862:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7932.901717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.912122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7932.932151] Lustre: Skipped 1 previous similar message [ 7932.950353] Lustre: Skipped 1 previous similar message [ 7938.018910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7938.033465] Lustre: Skipped 1 previous similar message [ 7938.467618] Lustre: server umount lustre-OST0000 complete [ 7944.680543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7944.694549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.269291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.289445] Lustre: Skipped 2 previous similar messages [ 7949.727514] Lustre: server umount lustre-MDT0000 complete [ 7953.389958] LustreError: 247918:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7953.408184] LustreError: 247918:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 7954.349211] LustreError: 247899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831342 with bad export cookie 6593475667757173794 [ 7954.363805] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7954.369059] LustreError: 247899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7961.293727] Lustre: server umount lustre-MDT0001 complete [ 7975.852580] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7978.507896] Key type lgssc unregistered [ 7978.790744] LNet: 253025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7978.796071] LNetError: 253025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7978.810143] LNet: Removed LNI 192.168.201.125@tcp [ 7979.681255] Key type .llcrypt unregistered [ 7979.684292] Key type ._llcrypt unregistered [ 8024.662196] Key type ._llcrypt registered [ 8024.664486] Key type .llcrypt registered [ 8024.772167] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8026.795820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8026.861081] alg: No test for adler32 (adler32-zlib) [ 8027.974525] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8028.327425] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8030.112177] Key type lgssc registered [ 8031.835391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8040.706459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8040.721792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.984296] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8042.008910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8042.164845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.091522] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8056.969884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8057.125490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8057.181697] Lustre: 255239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8057.556698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8062.792406] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8071.802311] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8078.538598] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8086.966057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8087.293660] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8087.451894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.385570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8093.439833] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8101.810878] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.184839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.230379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.308299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.313954] Lustre: Skipped 2 previous similar messages [ 8163.830702] Lustre: server umount lustre-OST0000 complete [ 8168.790806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8181.416076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8181.641242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.806414] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8188.839975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8188.847316] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8189.394972] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8197.960933] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.098663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8240.099948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.113686] LustreError: Skipped 1 previous similar message [ 8240.120552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8240.152470] Lustre: Skipped 2 previous similar messages [ 8245.393894] Lustre: server umount lustre-OST0000 complete [ 8250.338854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.341710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8250.350169] Lustre: Skipped 1 previous similar message [ 8250.364735] Lustre: Skipped 1 previous similar message [ 8255.457716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.466110] Lustre: Skipped 1 previous similar message [ 8256.269906] Lustre: server umount lustre-MDT0000 complete [ 8260.586493] LustreError: 255251:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.591820] LustreError: 255232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831648 with bad export cookie 6062363974014587137 [ 8260.598932] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.599016] LustreError: 255251:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 8260.618650] LustreError: 255232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8261.170955] Lustre: server umount lustre-MDT0001 complete [ 8274.000164] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8277.385935] Key type lgssc unregistered [ 8277.709885] LNet: 260666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8277.714697] LNetError: 260666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8278.762982] LNet: Removed LNI 192.168.201.125@tcp [ 8279.730567] Key type .llcrypt unregistered [ 8279.732823] Key type ._llcrypt unregistered [ 8306.837095] Key type ._llcrypt registered [ 8306.839581] Key type .llcrypt registered [ 8306.945504] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8308.334360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8308.482139] alg: No test for adler32 (adler32-zlib) [ 8309.628996] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8309.979822] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8311.728244] Key type lgssc registered [ 8312.952325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8321.076526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8321.117649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.741036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8327.333331] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8332.264126] LustreError: 262090:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.301443] LustreError: 262090:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8336.042629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.345617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.441563] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8347.269679] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8352.905860] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8358.747279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8359.055788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.362599] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8367.351378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8374.113439] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8402.401529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8402.408228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.423815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8403.426295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8403.433303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8403.433907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.446936] Lustre: Skipped 1 previous similar message [ 8406.058384] Lustre: server umount lustre-OST0000 complete [ 8413.154211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8413.164197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.187712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8416.210068] Lustre: server umount lustre-MDT0000 complete [ 8418.786684] LustreError: 262091:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.799890] LustreError: 262091:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8420.196590] LustreError: 262071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831808 with bad export cookie 16810215415113441936 [ 8420.199651] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8420.206219] LustreError: 262071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8420.615095] Lustre: server umount lustre-MDT0001 complete [ 8434.830450] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8438.240837] Key type lgssc unregistered [ 8438.528843] LNet: 266402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8438.536872] LNetError: 266402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8439.603594] LNet: Removed LNI 192.168.201.125@tcp [ 8440.976411] Key type .llcrypt unregistered [ 8440.979359] Key type ._llcrypt unregistered [ 8460.946756] Key type ._llcrypt registered [ 8460.948225] Key type .llcrypt registered [ 8461.094519] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8462.744925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8462.794163] alg: No test for adler32 (adler32-zlib) [ 8463.885376] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8464.212542] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8465.960208] Key type lgssc registered [ 8467.306939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8475.892713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8475.926747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.521457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8482.841962] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8486.895184] LustreError: 267623:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.908420] LustreError: 267623:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8490.990407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.477921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8496.435849] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8503.861867] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8509.624478] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.966911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.551949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.679467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8525.849172] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8535.125451] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8548.333457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8548.350799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.372658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8549.748531] Lustre: server umount lustre-OST0000 complete [ 8556.513834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.517069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.534802] Lustre: Skipped 1 previous similar message [ 8556.547098] Lustre: Skipped 1 previous similar message [ 8561.638870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.643644] Lustre: Skipped 1 previous similar message [ 8566.770448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.712468] Lustre: 266937:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831939/real 1787831939] req@000000008055b685 x1874677572072576/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787831955 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8567.759983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8568.288174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8568.717571] Lustre: server umount lustre-MDT0000 complete [ 8571.872109] Lustre: 266934:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831944/real 1787831944] req@000000001ffebdaa x1874677572074112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787831960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8571.897361] Lustre: 266934:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8572.884480] LustreError: 267603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831961 with bad export cookie 2303101133259204439 [ 8572.902747] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8572.904799] LustreError: 267603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8573.269977] Lustre: server umount lustre-MDT0001 complete [ 8585.946960] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8589.247217] Key type lgssc unregistered [ 8589.561520] LNet: 271306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8589.573988] LNetError: 271306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8589.586412] LNet: Removed LNI 192.168.201.125@tcp [ 8590.667429] Key type .llcrypt unregistered [ 8590.672620] Key type ._llcrypt unregistered [ 8617.927338] Key type ._llcrypt registered [ 8617.928881] Key type .llcrypt registered [ 8618.095622] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8619.449523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8619.687881] alg: No test for adler32 (adler32-zlib) [ 8620.787912] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8621.221939] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8622.978578] Key type lgssc registered [ 8624.604517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8632.657295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8632.685163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.046259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8637.556792] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8644.578769] LustreError: 272725:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8644.619994] LustreError: 272725:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8645.839947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.313359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8650.833203] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8658.618751] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8665.154309] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8671.858781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.156312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.736333] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8679.362437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8685.272332] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8717.285078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.294552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.303254] Lustre: Skipped 1 previous similar message [ 8717.304792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8717.313399] Lustre: Skipped 1 previous similar message [ 8721.982363] Lustre: server umount lustre-OST0000 complete [ 8727.522565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.536300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8732.641490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8732.658728] Lustre: Skipped 2 previous similar messages [ 8732.866886] Lustre: server umount lustre-MDT0000 complete [ 8736.540947] LustreError: 272705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832124 with bad export cookie 5247835255448541033 [ 8736.544929] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8736.549122] LustreError: 272705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8736.831896] Lustre: server umount lustre-MDT0001 complete [ 8746.430110] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8749.110428] Key type lgssc unregistered [ 8749.403714] LNet: 277048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8749.410032] LNetError: 277048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8749.433896] LNet: Removed LNI 192.168.201.125@tcp [ 8750.263459] Key type .llcrypt unregistered [ 8750.267479] Key type ._llcrypt unregistered [ 8769.167586] Key type ._llcrypt registered [ 8769.175336] Key type .llcrypt registered [ 8769.284168] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8770.247779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8770.319658] alg: No test for adler32 (adler32-zlib) [ 8771.598827] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8771.881856] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8773.528277] Key type lgssc registered [ 8774.538491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8784.230845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8784.257941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.040381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.515712] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8796.131920] LustreError: 278283:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.140400] LustreError: 278283:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8801.191761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.249272] LustreError: 278285:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8801.532227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8806.602839] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8815.560088] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8822.066806] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8828.055514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8828.537843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.615460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8834.696973] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8842.094796] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8861.664990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.667644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.676246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8861.683241] Lustre: Skipped 1 previous similar message [ 8861.703033] Lustre: Skipped 1 previous similar message [ 8865.437495] Lustre: server umount lustre-OST0000 complete [ 8871.909024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.916195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8871.934624] Lustre: Skipped 1 previous similar message [ 8871.954861] Lustre: Skipped 1 previous similar message [ 8877.025895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.148737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.159473] Lustre: Skipped 2 previous similar messages [ 8885.217745] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8885.540466] Lustre: server umount lustre-MDT0000 complete [ 8888.301580] LustreError: 278270:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8888.323525] LustreError: 278270:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8889.039171] LustreError: 278250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832277 with bad export cookie 13362409636341562107 [ 8889.060033] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.430889] Lustre: server umount lustre-MDT0001 complete [ 8901.141653] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8903.657283] Key type lgssc unregistered [ 8903.971514] LNet: 281987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8903.991597] LNetError: 281987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8904.014894] LNet: Removed LNI 192.168.201.125@tcp [ 8904.931174] Key type .llcrypt unregistered [ 8904.933171] Key type ._llcrypt unregistered [ 8916.972267] Key type ._llcrypt registered [ 8916.974933] Key type .llcrypt registered [ 8917.037202] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8934.501996] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8951.651957] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8952.969636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8953.113034] alg: No test for adler32 (adler32-zlib) [ 8954.153342] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8954.468291] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8956.192178] Key type lgssc registered [ 8957.474659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8994.995716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8995.031033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.365807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8996.448571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8996.596467] Lustre: lustre-MDT0000: new disk, initializing [ 8996.715776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8996.731160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9001.295073] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9012.217842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.345532] Lustre: 285543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9012.395235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9012.398688] Lustre: Skipped 1 previous similar message [ 9012.472578] Lustre: lustre-MDT0001: new disk, initializing [ 9012.628461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9012.705128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9012.726813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9018.311435] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9027.970220] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9033.829513] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9040.201946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9040.490466] Lustre: lustre-OST0000: new disk, initializing [ 9040.492934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9040.497460] Lustre: 287512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9040.553336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9046.719750] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9049.708755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9049.726033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9049.816173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9056.778043] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9062.407140] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9062.683508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9067.039600] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9067.260411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9070.565135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.566890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9070.572700] Lustre: Skipped 1 previous similar message [ 9070.581165] Lustre: Skipped 1 previous similar message [ 9075.686406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9080.804789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9080.808102] Lustre: Skipped 2 previous similar messages [ 9083.360136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9083.568044] Lustre: server umount lustre-OST0000 complete [ 9089.505995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9089.523738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9089.553152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9091.044550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.168894] Lustre: server umount lustre-MDT0000 complete [ 9099.274863] LustreError: 285534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832487 with bad export cookie 2154881863987427744 [ 9099.283381] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9099.295236] LustreError: 285534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9099.696786] Lustre: server umount lustre-MDT0001 complete [ 9109.076895] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:08:16 (1787832496) [ 9117.676366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.582157] LustreError: 289926:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9118.627458] LustreError: 289926:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9118.726845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9123.809341] LustreError: 289927:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.369388] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9127.919137] LustreError: 289926:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.027983] LustreError: 289927:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.614720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9134.106919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9139.316652] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9146.268584] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9152.273686] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9159.867484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9160.062900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9168.087369] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9180.536737] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9193.953386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9193.960052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.966672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9193.970984] Lustre: Skipped 2 previous similar messages [ 9194.979671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9199.888948] Lustre: server umount lustre-OST0000 complete [ 9205.218490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9205.221300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9205.229342] Lustre: Skipped 2 previous similar messages [ 9205.250822] Lustre: Skipped 3 previous similar messages [ 9209.964375] Lustre: server umount lustre-MDT0000 complete [ 9210.337290] LustreError: 289927:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9210.360917] LustreError: 289927:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9214.153730] LustreError: 289906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832602 with bad export cookie 2154881863987428626 [ 9214.158690] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.160415] LustreError: 289906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9214.616497] Lustre: server umount lustre-MDT0001 complete [ 9226.561623] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9229.373589] Key type lgssc unregistered [ 9229.724596] LNet: 293485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9229.739254] LNetError: 293485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9229.754534] LNet: Removed LNI 192.168.201.125@tcp [ 9231.005875] Key type .llcrypt unregistered [ 9231.007480] Key type ._llcrypt unregistered [ 9264.173328] Key type ._llcrypt registered [ 9264.175446] Key type .llcrypt registered [ 9264.274398] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9280.852409] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9282.260744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9282.288107] alg: No test for adler32 (adler32-zlib) [ 9283.422194] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 9283.767134] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9285.464551] Key type lgssc registered [ 9286.993262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9343.239323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9343.273524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9344.451591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9344.480553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9344.567346] Lustre: lustre-MDT0000: new disk, initializing [ 9344.721274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9344.775256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9348.446644] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9360.106714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.187911] Lustre: 297317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9360.226382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9360.228811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9360.233976] Lustre: Skipped 1 previous similar message [ 9360.296375] Lustre: lustre-MDT0001: new disk, initializing [ 9360.368225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9360.405787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9360.415692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9365.391517] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9375.091375] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9381.072518] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9387.434840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.772676] Lustre: lustre-OST2710: new disk, initializing [ 9387.776793] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9387.782807] Lustre: 299286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9387.855237] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9393.233342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9393.242484] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9393.338983] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9394.291506] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9403.682186] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9409.876555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9410.177693] Lustre: lustre-OST03e8: new disk, initializing [ 9410.189664] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9410.210211] Lustre: 300352:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9410.352663] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9415.865492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9415.884938] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9416.070283] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9417.473910] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9425.321812] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9433.143232] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9433.338278] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9437.679838] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9437.870919] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9443.445535] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9443.784913] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9449.162371] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9449.406184] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9456.504636] Lustre: server umount lustre-MDT0000 complete [ 9457.140609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9457.163836] Lustre: Skipped 1 previous similar message [ 9460.180137] LustreError: 297309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832848 with bad export cookie 6341857973284911094 [ 9460.192278] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9460.200981] LustreError: 297309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9460.699309] Lustre: server umount lustre-MDT0001 complete [ 9474.469953] Lustre: server umount lustre-OST2710 complete [ 9488.969067] Lustre: server umount lustre-OST03e8 complete [ 9506.913865] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9516.585194] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9547.435359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9547.615726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9547.641296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9547.724299] Lustre: lustre-MDT0000: new disk, initializing [ 9547.814395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9547.831383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9552.324181] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9563.380506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.481585] Lustre: 305601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9563.535579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9563.539379] Lustre: Skipped 1 previous similar message [ 9563.605835] Lustre: lustre-MDT0001: new disk, initializing [ 9563.697185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9563.748213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9563.762592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9568.807367] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9577.479419] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9582.981165] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.917373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.411162] Lustre: lustre-OST0000: new disk, initializing [ 9591.414858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9591.421505] Lustre: 307569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.571456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.052063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9597.075442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9597.250895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9601.513396] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9614.575991] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9620.271249] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9620.458688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9625.525842] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9625.732050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9628.137352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.150237] Lustre: Skipped 2 previous similar messages [ 9628.155686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9628.159260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9633.252184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9633.254861] Lustre: Skipped 1 previous similar message [ 9633.825115] Lustre: server umount lustre-OST0000 complete [ 9638.386984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9638.395750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9638.402871] Lustre: Skipped 2 previous similar messages [ 9638.413247] Lustre: Skipped 2 previous similar messages [ 9643.489786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9643.498522] Lustre: Skipped 1 previous similar message [ 9643.858526] Lustre: server umount lustre-MDT0000 complete [ 9647.919554] LustreError: 305594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833036 with bad export cookie 6341857973284912774 [ 9647.932929] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9648.459916] Lustre: server umount lustre-MDT0001 complete [ 9656.744211] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:17:24 (1787833044) [ 9658.069456] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9659.879670] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:17:27 (1787833047) [ 9662.966225] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9694.934440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.144296] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9695.163534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9695.364239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.242592] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9709.276270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9709.443551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9709.482974] Lustre: 311421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9715.237872] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9738.107663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.487260] Lustre: Denying initial registration attempt from nid 192.168.201.125@tcp, specified as failover [ 9738.502524] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9738.515751] LustreError: 312877:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9738.530517] LustreError: 312877:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9738.549968] LustreError: 312877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9738.656929] Lustre: server umount lustre-OST0000 complete [ 9738.662514] LustreError: 312877:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9745.894367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9745.919371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.950397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9751.437891] Lustre: server umount lustre-MDT0000 complete [ 9752.549323] LustreError: 311434:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.601843] LustreError: 311434:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 9756.443614] LustreError: 311415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833144 with bad export cookie 6341857973284913649 [ 9756.452693] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9756.460771] LustreError: 311415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9757.240477] Lustre: server umount lustre-MDT0001 complete [ 9769.753386] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9773.558786] Key type lgssc unregistered [ 9773.957792] LNet: 314000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9773.972819] LNetError: 314000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9775.031721] LNet: Removed LNI 192.168.201.125@tcp [ 9776.793643] Key type .llcrypt unregistered [ 9776.795312] Key type ._llcrypt unregistered [ 9790.913384] Key type ._llcrypt registered [ 9790.914864] Key type .llcrypt registered [ 9791.005727] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9791.891155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9791.934847] alg: No test for adler32 (adler32-zlib) [ 9793.100549] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 9793.280401] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9795.056529] Key type lgssc registered [ 9796.497093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9846.899949] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9856.446721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9856.456835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.848953] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9857.876594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9858.055607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9862.994085] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9871.022691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.090132] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9871.112364] Lustre: 316765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9871.330487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9876.735685] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9884.220456] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9889.927150] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9910.184242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9910.474791] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9910.762444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.695402] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9925.664380] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9928.162377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9928.174388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.181282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9928.207151] Lustre: Skipped 1 previous similar message [ 9933.304534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9933.315050] Lustre: Skipped 2 previous similar messages [ 9934.171344] Lustre: server umount lustre-OST0000 complete [ 9938.401947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.417063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9938.422415] Lustre: Skipped 1 previous similar message [ 9943.522690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9943.533231] Lustre: Skipped 1 previous similar message [ 9944.673025] Lustre: server umount lustre-MDT0000 complete [ 9948.967478] LustreError: 316757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833337 with bad export cookie 4001429309785093756 [ 9948.978910] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.007283] LustreError: 316757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9949.464291] Lustre: server umount lustre-MDT0001 complete [ 9960.095613] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9964.160249] Key type lgssc unregistered [ 9964.645307] LNet: 320344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9964.650903] LNetError: 320344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9965.669847] LNet: Removed LNI 192.168.201.125@tcp [ 9967.041454] Key type .llcrypt unregistered [ 9967.044798] Key type ._llcrypt unregistered [ 9994.636283] Key type ._llcrypt registered [ 9994.639246] Key type .llcrypt registered [ 9994.734087] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9996.061358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9996.105056] alg: No test for adler32 (adler32-zlib) [ 9997.253340] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 9997.518338] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9999.264209] Key type lgssc registered [10000.527226] Lustre: Echo OBD driver; http://www.lustre.org/ [10007.784773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10007.799763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.249155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10014.057447] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10018.793967] LustreError: 321765:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10018.814640] LustreError: 321765:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10022.669860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.243984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10028.351351] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10036.955124] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10042.331941] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10051.565896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10052.095742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10059.078946] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10066.067971] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10078.689707] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10078.710474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10078.730721] Lustre: Skipped 1 previous similar message [10078.776874] Lustre: Skipped 2 previous similar messages [10082.722946] Lustre: server umount lustre-MDT0000 complete [10083.824263] LustreError: 321764:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.854289] LustreError: 321764:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10087.334668] LustreError: 321745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833475 with bad export cookie 17225570686355095046 [10087.339109] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.354974] LustreError: 321745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10088.931508] LustreError: 321761:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.937346] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10088.944045] LustreError: 321761:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10088.964990] Lustre: Skipped 1 previous similar message [10088.970448] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10088.976383] Lustre: Skipped 1 previous similar message [10092.983540] LustreError: 323682:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.983808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10093.031894] LustreError: 323682:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10093.050385] Lustre: Skipped 1 previous similar message [10093.966852] Lustre: server umount lustre-MDT0001 complete [10100.014324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10109.498319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.948150] LustreError: 323597:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.053108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.156716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10117.938703] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10122.209443] LustreError: 323597:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.243153] LustreError: 323597:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10123.254817] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10127.302077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.743758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10132.465151] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10132.985040] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10139.721484] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10141.263831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10147.052661] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10148.842109] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10153.953266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10153.961684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10153.979511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.561707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.571836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10158.576820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.288330] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10168.545262] Lustre: server umount lustre-OST0000 complete [10173.923280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10173.924088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10173.944518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10173.961219] Lustre: Skipped 4 previous similar messages [10178.854698] Lustre: server umount lustre-MDT0000 complete [10179.042310] LustreError: 325361:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10179.058774] LustreError: 325361:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10182.555664] LustreError: 321744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833570 with bad export cookie 17225570686355102564 [10182.559071] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10182.565281] LustreError: 321744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10183.020245] Lustre: server umount lustre-MDT0001 complete [10194.675530] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [10197.675617] Key type lgssc unregistered [10198.006553] LNet: 328033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10198.022278] LNetError: 328033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10198.076191] LNet: Removed LNI 192.168.201.125@tcp [10199.088893] Key type .llcrypt unregistered [10199.100853] Key type ._llcrypt unregistered [10226.569310] Key type ._llcrypt registered [10226.572125] Key type .llcrypt registered [10226.643124] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [10227.900617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10228.212916] alg: No test for adler32 (adler32-zlib) [10229.368701] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [10229.627000] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [10231.336217] Key type lgssc registered [10232.676982] Lustre: Echo OBD driver; http://www.lustre.org/ [10239.030400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10239.048414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10240.362759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10243.966789] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10250.722898] LustreError: 329451:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.729773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10250.736769] LustreError: 329451:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10251.069362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10255.070591] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10262.235991] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10267.983512] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10270.691927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.705503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.706037] Lustre: Skipped 1 previous similar message [10270.713834] Lustre: Skipped 1 previous similar message [10275.810111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.821855] Lustre: Skipped 1 previous similar message [10276.835291] LustreError: 329452:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.837157] Lustre: server umount lustre-MDT0000 complete [10276.848923] LustreError: 329452:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10280.771133] LustreError: 329432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833668 with bad export cookie 17735195015887916101 [10280.781553] LustreError: 329432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10280.784380] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10281.341205] Lustre: server umount lustre-MDT0001 complete [10294.876948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10295.143057] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10295.179470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10295.394096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10299.713546] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10308.634782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.761490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10308.796913] Lustre: 332040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10309.108327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10314.119216] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10323.096618] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10329.907360] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10339.451424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10339.737160] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10339.751453] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10339.760714] LustreError: 333924:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10339.765550] LustreError: 333924:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10339.768703] LustreError: 333924:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10339.852669] Lustre: server umount lustre-OST0000 complete [10339.855486] LustreError: 333924:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10350.124791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10350.223723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10350.296632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10356.178737] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10356.884376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10363.770350] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10372.199819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10372.316377] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10372.392912] Lustre: lustre-OST0001: new disk, initializing [10372.399877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10372.406926] Lustre: 335466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10372.505188] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10378.197600] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10379.450146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10379.477785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10379.558946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10386.983343] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10389.991399] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10390.005165] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10390.012791] LustreError: Skipped 1 previous similar message [10390.019446] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10395.110109] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10395.126987] Lustre: Skipped 2 previous similar messages [10400.225429] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10400.235719] Lustre: Skipped 1 previous similar message [10404.320457] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10404.468536] Lustre: server umount lustre-OST0001 complete [10405.363338] LustreError: 336269:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.383469] LustreError: 336269:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10408.421259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10408.443342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.468791] Lustre: Skipped 1 previous similar message [10408.474965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10410.474684] LustreError: 334392:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10410.477204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10410.490461] LustreError: 334392:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10414.331724] Lustre: server umount lustre-OST0000 complete [10420.711108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.517809] Lustre: server umount lustre-MDT0000 complete [10425.832507] LustreError: 332049:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10425.854590] LustreError: 332049:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10428.610496] LustreError: 332779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833816 with bad export cookie 17735195015887916647 [10428.621084] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10428.623815] LustreError: 332779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10428.889443] Lustre: server umount lustre-MDT0001 complete [10439.072626] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [10441.691694] Key type lgssc unregistered [10441.971278] LNet: 337426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10441.976078] LNetError: 337426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10442.003237] LNet: Removed LNI 192.168.201.125@tcp [10442.893360] Key type .llcrypt unregistered [10442.896987] Key type ._llcrypt unregistered [10510.994156] Key type ._llcrypt registered [10510.995706] Key type .llcrypt registered [10511.097442] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [10526.060797] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [10527.740136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10527.796853] alg: No test for adler32 (adler32-zlib) [10528.856289] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [10529.091897] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [10530.768350] Key type lgssc registered [10532.235655] Lustre: Echo OBD driver; http://www.lustre.org/ [10567.174609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10567.205435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.620890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10568.677453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10568.821968] Lustre: lustre-MDT0000: new disk, initializing [10568.909765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10568.927291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10573.547456] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10584.755174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10584.857496] Lustre: 342362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10584.909896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10584.914291] Lustre: Skipped 1 previous similar message [10584.982502] Lustre: lustre-MDT0001: new disk, initializing [10585.076558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10585.126423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10585.136651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10590.007977] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10599.457521] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10604.683807] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10610.742141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10611.070664] Lustre: lustre-OST0000: new disk, initializing [10611.079676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10611.092639] Lustre: 344330:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10611.209766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10617.126974] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10620.517803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10620.534758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10620.675470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10626.302970] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10631.136946] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10631.389432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10636.205362] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10636.399899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10641.378053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10641.378068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.389922] LustreError: Skipped 1 previous similar message [10641.409938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10644.554429] Lustre: server umount lustre-OST0000 complete [10651.622566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10651.630312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10651.638117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10651.638125] Lustre: Skipped 1 previous similar message [10651.672825] Lustre: Skipped 2 previous similar messages [10656.738109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10656.744273] Lustre: Skipped 2 previous similar messages [10660.834678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10662.368807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10662.574693] Lustre: server umount lustre-MDT0000 complete [10666.035339] LustreError: 344329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834054 with bad export cookie 10110344108797941505 [10666.042433] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10666.044209] LustreError: 344329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10666.540812] Lustre: server umount lustre-MDT0001 complete [10675.894910] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:34:23 (1787834063) [10683.700704] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:34:31 (1787834071) [10692.108378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.530965] LustreError: 347085:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.557344] LustreError: 347085:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10692.648915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10697.457969] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10697.717698] LustreError: 347086:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10701.822244] LustreError: 347085:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.641031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.918242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10711.369908] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10719.099383] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10724.500533] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10730.912591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10731.256145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10737.508528] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10744.845387] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10752.137823] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10754.481617] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10756.398387] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10758.324558] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10760.367085] Lustre: Failing over lustre-MDT0000 [10760.666357] Lustre: server umount lustre-MDT0000 complete [10762.191393] LustreError: 347081:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10763.238659] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.243076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10763.251195] Lustre: Skipped 1 previous similar message [10767.288823] LustreError: 347082:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.306254] LustreError: 347082:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10777.535777] LustreError: 347082:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.550249] LustreError: 347082:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [10778.875852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10778.952772] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10779.214533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10779.282927] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10783.666563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10784.261219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10784.300672] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10784.334309] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10784.360111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10793.779918] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10795.283607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10799.004423] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10801.290336] Lustre: Failing over lustre-MDT0000 [10801.684499] Lustre: server umount lustre-MDT0000 complete [10804.198770] LustreError: 347080:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10804.221783] LustreError: 347080:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10804.707847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10804.708063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.067239] Lustre: 340581:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834192/real 1787834192] req@00000000801cd177 x1874679737441280/t0(0) o400->MGC192.168.201.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787834208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10820.100388] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10821.719174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10830.307038] LustreError: 340580:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002ebeb6c7 x1874679737445760/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10830.576893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10830.622340] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10830.788191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10835.469941] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10835.957150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10835.964609] Lustre: Skipped 2 previous similar messages [10836.010580] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [10836.042578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10845.782463] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10847.183496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10856.417771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10856.423060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.432529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.432546] Lustre: Skipped 1 previous similar message [10856.478776] Lustre: Skipped 4 previous similar messages [10859.025357] Lustre: server umount lustre-MDT0000 complete [10861.538911] LustreError: 347086:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.556042] LustreError: 347086:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 24 previous similar messages [10862.912413] LustreError: 347066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834251 with bad export cookie 10110344108797944221 [10862.918335] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10862.925613] LustreError: 347066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10863.463733] Lustre: server umount lustre-MDT0001 complete [10877.983452] Lustre: server umount lustre-OST0000 complete [10893.676046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.982705] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10908.122248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10908.397380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10908.407062] Lustre: Skipped 1 previous similar message [10913.972752] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10922.497109] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10929.786468] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10937.489942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10938.926192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10945.093268] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10953.403610] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10959.339501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10959.355326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10959.380031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10959.388547] Lustre: Skipped 2 previous similar messages [10965.724794] Lustre: server umount lustre-OST0000 complete [10976.411241] Lustre: server umount lustre-MDT0000 complete [10978.793845] LustreError: 356009:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10978.822123] LustreError: 356009:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10980.395156] LustreError: 353410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834368 with bad export cookie 10110344108797944970 [10980.402900] LustreError: 353410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10980.405199] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10980.727313] Lustre: server umount lustre-MDT0001 complete [10993.560426] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [10996.432166] Key type lgssc unregistered [10996.721386] LNet: 356965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10996.728876] LNetError: 356965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10997.806814] LNet: Removed LNI 192.168.201.125@tcp [10999.056101] Key type .llcrypt unregistered [10999.059594] Key type ._llcrypt unregistered [11024.048275] Key type ._llcrypt registered [11024.050063] Key type .llcrypt registered [11024.153328] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [11025.035611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11025.184825] alg: No test for adler32 (adler32-zlib) [11026.209649] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [11026.379856] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [11028.011443] Key type lgssc registered [11029.029231] Lustre: Echo OBD driver; http://www.lustre.org/ [11036.562653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11036.579148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.951866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11042.838188] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11048.419445] LustreError: 358481:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.439800] LustreError: 358481:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11051.364292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11051.919947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11056.584862] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11063.783460] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11069.241703] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11076.794087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.348906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11083.517509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11084.006090] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11091.134339] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11103.201615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11103.212444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11103.236535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11104.230292] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.240903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11104.267043] LustreError: 360364:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.287533] LustreError: 360364:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11104.378814] Lustre: server umount lustre-MDT0000 complete [11108.312967] LustreError: 358461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834496 with bad export cookie 15068463834170566783 [11108.316113] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11108.334572] LustreError: 358461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11108.734527] Lustre: server umount lustre-MDT0001 complete [11122.232881] Lustre: server umount lustre-OST0000 complete [11135.119754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.419858] LustreError: 362191:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11135.481960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11139.793332] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11140.578801] LustreError: 362192:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.701211] LustreError: 362191:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.205033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11149.616575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11155.492277] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11165.202786] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11170.633975] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11178.411509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11178.700483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11180.740377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11185.042856] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11192.035571] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11204.580745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.587021] Lustre: Skipped 1 previous similar message [11204.599785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11204.605413] Lustre: Skipped 2 previous similar messages [11209.580949] Lustre: server umount lustre-MDT0000 complete [11209.713341] LustreError: 364447:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.739615] LustreError: 364447:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11213.844252] LustreError: 362172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834602 with bad export cookie 15068463834170567896 [11213.872137] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11214.298936] Lustre: server umount lustre-MDT0001 complete [11228.987264] Lustre: server umount lustre-OST0000 complete [11244.800968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.253212] LustreError: 366106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.338701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11250.144328] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11258.666273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.998389] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11271.425270] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11276.576297] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11283.399424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.736366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11283.743633] Lustre: Skipped 1 previous similar message [11289.962791] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11290.925870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11297.604564] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11304.529479] Lustre: server umount lustre-OST0000 complete [11310.561557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11310.578884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11310.608989] Lustre: Skipped 2 previous similar messages [11310.623196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11316.706316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11316.721395] Lustre: Skipped 3 previous similar messages [11318.854936] Lustre: server umount lustre-MDT0000 complete [11320.800884] Lustre: 357794:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834693/real 1787834693] req@000000005e9c70f4 x1874680258581760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787834709 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11320.828339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11320.842473] Lustre: Skipped 1 previous similar message [11320.850192] LustreError: 366106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.867213] LustreError: 366106:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11321.825133] Lustre: 357792:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834694/real 1787834694] req@00000000f117fb2d x1874680258582528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787834710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11322.521962] LustreError: 366087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834710 with bad export cookie 15068463834170568911 [11322.526407] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11322.546820] LustreError: 366087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11323.183994] Lustre: server umount lustre-MDT0001 complete [11335.382550] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [11339.201525] Key type lgssc unregistered [11339.467737] LNet: 369640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11339.478569] LNetError: 369640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11339.520438] LNet: Removed LNI 192.168.201.125@tcp [11340.423276] Key type .llcrypt unregistered [11340.426328] Key type ._llcrypt unregistered [11366.636241] Key type ._llcrypt registered [11366.641740] Key type .llcrypt registered [11366.778795] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [11368.162644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11368.220078] alg: No test for adler32 (adler32-zlib) [11369.324198] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [11369.740770] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [11371.464175] Key type lgssc registered [11372.560432] Lustre: Echo OBD driver; http://www.lustre.org/ [11379.436771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11379.461489] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.478353] LustreError: 371078:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11379.507161] LustreError: 371078:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11379.520747] LustreError: 371078:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11379.530318] LustreError: 371078:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11379.551159] LustreError: 371078:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11387.004222] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11387.017610] LustreError: 371475:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11387.058636] LustreError: 371475:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11387.066626] LustreError: 371475:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11387.072344] LustreError: 371475:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11387.075846] LustreError: 371475:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11408.183595] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [11411.090854] Key type lgssc unregistered [11411.295616] LNet: 372483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11411.301594] LNetError: 372483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11411.309792] LNet: Removed LNI 192.168.201.125@tcp [11412.038246] Key type .llcrypt unregistered [11412.040059] Key type ._llcrypt unregistered [11430.339793] Key type ._llcrypt registered [11430.343091] Key type .llcrypt registered [11430.530696] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [11443.869381] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [11444.931268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11445.133903] alg: No test for adler32 (adler32-zlib) [11446.203779] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [11446.410352] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [11448.074403] Key type lgssc registered [11449.795145] Lustre: Echo OBD driver; http://www.lustre.org/ [11485.159946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11485.194212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.557857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11486.589300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11486.645522] Lustre: lustre-MDT0000: new disk, initializing [11486.717631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11486.732785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11491.191351] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11501.796322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11501.937254] Lustre: 375634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11502.010042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11502.013803] Lustre: Skipped 1 previous similar message [11502.127708] Lustre: lustre-MDT0001: new disk, initializing [11502.252908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11502.315378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11502.333877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11507.475840] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11516.929340] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11522.222800] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11528.859899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11529.302376] Lustre: lustre-OST0000: new disk, initializing [11529.313678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11529.324255] Lustre: 377601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11529.404246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11535.720847] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11537.504916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11537.521834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11537.765599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11546.382302] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11552.194839] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11552.454632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11557.439969] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11557.665855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11563.505124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11563.519093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11563.521987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11563.531033] LustreError: Skipped 1 previous similar message [11563.548474] Lustre: Skipped 1 previous similar message [11568.612552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11568.615546] Lustre: Skipped 1 previous similar message [11573.731235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11573.737676] Lustre: Skipped 1 previous similar message [11574.240161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11574.521992] Lustre: server umount lustre-OST0000 complete [11578.857790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.865504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11578.869216] Lustre: Skipped 1 previous similar message [11578.894445] Lustre: Skipped 1 previous similar message [11583.975088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11584.395055] Lustre: server umount lustre-MDT0000 complete [11588.190687] LustreError: 375626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834976 with bad export cookie 11233781740998799047 [11588.204950] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11588.209083] LustreError: 375626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11588.517702] Lustre: server umount lustre-MDT0001 complete [11596.193109] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11597.833606] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:49:45 (1787834985) [11604.384993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.920964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11609.196632] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11610.081630] LustreError: 380065:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.097726] LustreError: 380065:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11615.208268] LustreError: 380064:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.700395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11617.013130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11621.553297] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11628.359638] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11633.627070] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11640.169784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11640.563673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11646.227836] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11652.891364] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11659.627292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11659.746299] Lustre: lustre-OST0001: new disk, initializing [11659.754441] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11659.760601] Lustre: 382900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11659.851915] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11665.940477] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11666.004653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11666.017352] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11666.126434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11675.097347] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11681.779347] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11681.782972] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.783966] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11681.783972] Lustre: Skipped 1 previous similar message [11681.796066] LustreError: Skipped 1 previous similar message [11681.845955] Lustre: Skipped 1 previous similar message [11685.330852] Lustre: server umount lustre-OST0001 complete [11686.886726] LustreError: 383268:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.901503] LustreError: 383268:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11692.021745] LustreError: 381883:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.044343] LustreError: 381883:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11697.151626] LustreError: 381885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.175814] LustreError: 381885:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11704.296067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11704.302320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.333743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11704.346764] Lustre: Skipped 2 previous similar messages [11707.368220] LustreError: 381884:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.405107] LustreError: 381884:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11709.740687] Lustre: server umount lustre-OST0000 complete [11714.529821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11714.545993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11714.565986] Lustre: Skipped 1 previous similar message [11720.359491] Lustre: server umount lustre-MDT0000 complete [11723.970087] LustreError: 380045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835112 with bad export cookie 11233781740998799929 [11723.983244] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11723.984680] LustreError: 380045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11730.707227] Lustre: server umount lustre-MDT0001 complete [11741.309229] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [11744.128743] Key type lgssc unregistered [11744.386720] LNet: 384891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11744.401440] LNetError: 384891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11744.428257] LNet: Removed LNI 192.168.201.125@tcp [11745.169622] Key type .llcrypt unregistered [11745.174165] Key type ._llcrypt unregistered [11797.237036] Key type ._llcrypt registered [11797.241840] Key type .llcrypt registered [11797.395314] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [11798.748862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11798.826648] alg: No test for adler32 (adler32-zlib) [11799.920724] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [11800.167182] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [11801.848883] Key type lgssc registered [11803.062667] Lustre: Echo OBD driver; http://www.lustre.org/ [11811.061230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11811.090291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.389136] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11812.416545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11812.619686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11817.624979] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11826.464938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.528424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11826.550487] Lustre: 387690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11826.805524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11830.749054] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11837.106204] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11843.966395] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11847.144036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11847.155753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11847.174116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11850.722714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11850.723376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11850.756548] Lustre: Skipped 1 previous similar message [11852.810053] Lustre: server umount lustre-MDT0000 complete [11856.901314] LustreError: 387684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835245 with bad export cookie 16305471025373949731 [11856.902755] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11856.912154] LustreError: 387684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11857.565965] Lustre: server umount lustre-MDT0001 complete [11869.641083] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:54:16 (1787835256) [11878.545403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11879.022935] LustreError: 390206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11879.061547] LustreError: 390206:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11879.174720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11884.524039] LustreError: 390207:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11885.413960] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11889.640947] LustreError: 390206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.753248] LustreError: 390207:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.363674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11895.698318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11900.285204] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11907.124686] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11911.992883] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11918.753279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11918.913809] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11919.071465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11925.042674] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11932.197579] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11939.965896] Lustre: Permanently deactivating lustre-OST0000 [11939.981270] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11948.577904] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11950.442926] LustreError: 393241:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11950.449147] LustreError: 393241:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11950.453377] LustreError: 393241:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11952.890281] LustreError: 393289:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11952.895992] LustreError: 393289:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11961.091513] Lustre: server umount lustre-OST0000 complete [11963.872818] LustreError: 393539:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11963.878432] LustreError: 393539:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11967.457565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11967.461209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11967.483895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11973.605250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11973.618965] Lustre: Skipped 3 previous similar messages [11975.875839] Lustre: server umount lustre-MDT0000 complete [11978.722528] LustreError: 390934:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.507096] LustreError: 390188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835367 with bad export cookie 16305471025373950312 [11979.511478] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11979.519315] LustreError: 390188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11989.203036] Lustre: server umount lustre-MDT0001 complete [11997.761088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12002.057659] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12004.486684] LustreError: 394698:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12010.158169] Lustre: 395037:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.125@tcp [12011.942743] Lustre: server umount MGS complete [12022.252546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.653538] LustreError: 395761:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.670084] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12022.672755] LustreError: 395761:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12022.683445] Lustre: Skipped 1 previous similar message [12022.758408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12027.446369] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12035.337695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12035.599336] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12040.180833] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12047.756381] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12053.909798] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12056.063638] Lustre: Permanently reactivating lustre-OST0000 [12056.074226] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12056.077757] Lustre: Skipped 2 previous similar messages [12082.144386] LustreError: 397493:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12088.283119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12088.674983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12088.682795] Lustre: Skipped 1 previous similar message [12094.567900] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12101.855633] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12107.233163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12107.237785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.248394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12107.264086] Lustre: Skipped 2 previous similar messages [12107.275596] Lustre: Skipped 1 previous similar message [12109.334897] Lustre: server umount lustre-OST0000 complete [12117.476224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12117.481957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.488911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.499105] Lustre: Skipped 1 previous similar message [12119.658262] Lustre: server umount lustre-MDT0000 complete [12122.596327] LustreError: 395756:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.614370] LustreError: 395756:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12123.000315] LustreError: 395743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835511 with bad export cookie 16305471025373951362 [12123.008815] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12123.405196] Lustre: server umount lustre-MDT0001 complete [12136.554748] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12139.132713] Key type lgssc unregistered [12139.463549] LNet: 400127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12139.467892] LNetError: 400127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12139.484684] LNet: Removed LNI 192.168.201.125@tcp [12140.525854] Key type .llcrypt unregistered [12140.528730] Key type ._llcrypt unregistered [12159.135361] Key type ._llcrypt registered [12159.140663] Key type .llcrypt registered [12159.244440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [12176.178915] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12177.386102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12177.576468] alg: No test for adler32 (adler32-zlib) [12178.630878] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [12178.958316] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [12180.680176] Key type lgssc registered [12182.115505] Lustre: Echo OBD driver; http://www.lustre.org/ [12215.078755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12215.112759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12216.423804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12216.478740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12216.568287] Lustre: lustre-MDT0000: new disk, initializing [12216.702244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12216.737935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12221.440113] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12231.805438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.939054] Lustre: 403284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12232.006755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12232.010387] Lustre: Skipped 1 previous similar message [12232.067234] Lustre: lustre-MDT0001: new disk, initializing [12232.144814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12232.176179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12232.181651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12237.148571] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12246.330581] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12250.915675] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12258.470076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12258.697886] Lustre: lustre-OST0000: new disk, initializing [12258.702468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12258.707835] Lustre: 405252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12258.790206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12265.331509] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12267.068250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12267.081834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12267.160360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12274.627927] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12279.572344] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12279.881265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12284.142753] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12284.465027] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12287.972399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12287.979012] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12287.982984] LustreError: Skipped 1 previous similar message [12287.983884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12287.996796] Lustre: Skipped 1 previous similar message [12288.019046] Lustre: Skipped 1 previous similar message [12292.418623] Lustre: server umount lustre-OST0000 complete [12298.213810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12298.218681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12298.232360] Lustre: Skipped 1 previous similar message [12298.243914] Lustre: Skipped 1 previous similar message [12302.059029] Lustre: server umount lustre-MDT0000 complete [12305.904886] LustreError: 403275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835694 with bad export cookie 4332337958318456517 [12305.913733] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.915051] LustreError: 403275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12306.444261] Lustre: server umount lustre-MDT0001 complete [12315.886574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12317.611287] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:01:45 (1787835705) [12325.561926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.434114] LustreError: 407722:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.462247] LustreError: 407722:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12326.556612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12332.010893] LustreError: 407723:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12332.422612] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12336.101073] LustreError: 407722:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12341.220514] LustreError: 407723:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.288753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12342.662739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12348.332273] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12356.597854] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12362.378436] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12368.796433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12369.210785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12376.395832] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12384.519876] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12386.400685] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12389.193679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12397.026421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12397.031844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.048178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12398.054672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12400.574786] Lustre: server umount lustre-OST0000 complete [12408.295935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.314091] Lustre: Skipped 1 previous similar message [12408.323751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12408.341386] Lustre: Skipped 3 previous similar messages [12411.074982] Lustre: server umount lustre-MDT0000 complete [12413.413275] LustreError: 407722:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12413.455200] LustreError: 407722:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12414.952563] LustreError: 407704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835803 with bad export cookie 4332337958318457399 [12414.956208] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12414.965194] LustreError: 407704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12415.302550] Lustre: server umount lustre-MDT0001 complete [12426.805625] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12429.795683] Key type lgssc unregistered [12430.190231] LNet: 411450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12430.202838] LNetError: 411450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12430.269832] LNet: Removed LNI 192.168.201.125@tcp [12430.973182] Key type .llcrypt unregistered [12430.978133] Key type ._llcrypt unregistered [12457.838270] Key type ._llcrypt registered [12457.840535] Key type .llcrypt registered [12457.952363] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12458.332484] Key type .llcrypt unregistered [12458.342359] Key type ._llcrypt unregistered [12477.630152] Key type ._llcrypt registered [12477.636110] Key type .llcrypt registered [12477.795421] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12478.930218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12479.107126] alg: No test for adler32 (adler32-zlib) [12480.378102] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [12480.714808] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [12482.576251] Key type lgssc registered [12483.838955] Lustre: Echo OBD driver; http://www.lustre.org/ [12491.598348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12491.621978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.976964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12497.523992] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12503.532507] LustreError: 413666:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.564601] LustreError: 413666:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12504.550473] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.060080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12507.587831] LustreError: 413666:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.617762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12507.643730] LustreError: 413666:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12512.740434] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.760689] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12516.696111] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12517.858775] LustreError: 413666:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.868524] LustreError: 413666:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12518.912343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12525.282844] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12529.123510] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12529.148131] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [12534.217506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.707710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12541.188361] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12552.451529] Lustre: server umount lustre-OST0000 complete [12557.804191] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12557.826420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.851645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12563.432517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12563.437481] Lustre: Skipped 3 previous similar messages [12566.623914] Lustre: server umount lustre-MDT0000 complete [12568.546720] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12568.561070] LustreError: 413667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12569.569560] Lustre: 412979:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835941/real 1787835941] req@00000000e5f610fe x1874681783153152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787835957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12569.620442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12569.652354] Lustre: Skipped 1 previous similar message [12571.487948] LustreError: 413647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835959 with bad export cookie 11911519375708587210 [12571.511746] LustreError: 413647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12571.513350] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12572.173933] Lustre: server umount lustre-MDT0001 complete [12583.885747] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12587.669914] Key type lgssc unregistered [12588.267470] LNet: 416821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12588.277148] LNetError: 416821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12588.332499] LNet: Removed LNI 192.168.201.125@tcp [12589.933846] Key type .llcrypt unregistered [12589.945479] Key type ._llcrypt unregistered [12620.807327] Key type ._llcrypt registered [12620.808707] Key type .llcrypt registered [12621.040955] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12622.570756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12622.680688] alg: No test for adler32 (adler32-zlib) [12623.902827] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [12624.095927] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [12625.776176] Key type lgssc registered [12627.144486] Lustre: Echo OBD driver; http://www.lustre.org/ [12634.810922] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12634.827847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12650.336452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12655.456301] LustreError: 418214:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [12682.208906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12688.683414] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12696.713603] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12703.730737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12703.925594] LustreError: 417550:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c1e8ee49 x1874681934121216/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12704.392903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12704.447561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12709.870805] LustreError: 418239:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.895490] LustreError: 418239:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12709.956217] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12714.985781] LustreError: 418240:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12717.034811] LustreError: 419252:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12717.056164] LustreError: 419252:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12718.991135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12719.323267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12724.388883] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12732.987692] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12739.188751] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12749.057959] Lustre: server umount lustre-OST0000 complete [12754.920314] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12754.931066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.961315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12755.431051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12760.546659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12760.559177] Lustre: Skipped 3 previous similar messages [12763.691322] LustreError: 421033:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000687b0717) refcount nonzero (1) after lock cleanup; forcing cleanup. [12763.749666] Lustre: server umount lustre-MDT0000 complete [12765.669929] LustreError: 419257:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.700058] LustreError: 419257:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12766.038232] Lustre: 417554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836138/real 1787836138] req@0000000046cf9307 x1874681934149632/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787836154 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12766.091726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12768.800217] LustreError: 418220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836157 with bad export cookie 14069663893888083153 [12768.805684] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12768.818383] LustreError: 418220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12769.159532] Lustre: server umount lustre-MDT0001 complete [12782.964994] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12786.907269] Key type lgssc unregistered [12787.134711] LNet: 421788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12787.141979] LNetError: 421788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12787.158789] LNet: Removed LNI 192.168.201.125@tcp [12787.967207] Key type .llcrypt unregistered [12787.969323] Key type ._llcrypt unregistered [12816.558682] Key type ._llcrypt registered [12816.560132] Key type .llcrypt registered [12816.604985] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12817.315505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12817.339350] alg: No test for adler32 (adler32-zlib) [12818.401778] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [12818.623699] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [12820.320219] Key type lgssc registered [12821.757238] Lustre: Echo OBD driver; http://www.lustre.org/ [12830.145855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12830.169909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12831.703937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12836.591270] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12840.933785] LustreError: 423209:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.965041] LustreError: 423209:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12846.050446] LustreError: 423210:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.343723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12846.833427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12852.218258] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12860.267825] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12865.900594] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12873.170857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12873.775755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12875.835118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12881.164244] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12889.428023] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12893.157233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12893.165460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12893.184480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12896.199482] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) [12896.227921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12896.235497] Lustre: Skipped 1 previous similar message [12899.545122] Lustre: server umount lustre-MDT0000 complete [12901.295348] LustreError: 423206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12902.528209] LustreError: 423191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.25@tcp arrived at 1787836290 with bad export cookie 3582410779497955818 [12904.944687] LustreError: 423191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836293 with bad export cookie 3582410779497954929 [12904.959868] LustreError: 423191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12904.976503] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12906.465118] LustreError: 424193:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12906.466170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12906.487172] LustreError: 424193:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [12906.515623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12906.521450] Lustre: Skipped 3 previous similar messages [12909.474128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12911.239470] Lustre: server umount lustre-OST0000 complete [12914.659146] LustreError: 424193:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.697491] LustreError: 424193:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12920.631083] Lustre: server umount lustre-MDT0001 complete [12934.269767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12937.684374] Key type lgssc unregistered [12937.857792] LNet: 426899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12937.862802] LNetError: 426899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12937.875061] LNet: Removed LNI 192.168.201.125@tcp [12938.583067] Key type .llcrypt unregistered [12938.584854] Key type ._llcrypt unregistered [12966.342940] Key type ._llcrypt registered [12966.346672] Key type .llcrypt registered [12966.562411] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12967.926619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12968.134887] alg: No test for adler32 (adler32-zlib) [12969.301151] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [12969.601836] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [12971.308628] Key type lgssc registered [12972.782428] Lustre: Echo OBD driver; http://www.lustre.org/ [12981.715738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12981.732259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12983.319617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12987.901684] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12993.511199] LustreError: 428319:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12993.516223] LustreError: 428319:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12997.979296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12998.260520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13003.791919] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13011.939569] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13018.259310] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13025.589783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13025.934170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13028.020419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13032.825080] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13041.597900] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13048.803366] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.810324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13048.825619] Lustre: Skipped 1 previous similar message [13048.851280] Lustre: Skipped 1 previous similar message [13049.832023] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [13053.389593] Lustre: server umount lustre-MDT0001 complete [13053.935992] LustreError: 428313:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13053.969981] LustreError: 428313:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13059.062895] LustreError: 428318:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13059.084575] LustreError: 428318:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13064.162540] LustreError: 428319:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.166552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.167510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13064.252921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13067.050163] Lustre: server umount lustre-OST0000 complete [13069.306430] LustreError: 428318:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13072.052219] Lustre: server umount lustre-MDT0000 complete [13088.933783] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [13093.095161] Key type lgssc unregistered [13093.472667] LNet: 432007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13093.477385] LNetError: 432007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13094.504808] LNet: Removed LNI 192.168.201.125@tcp [13095.813229] Key type .llcrypt unregistered [13095.815907] Key type ._llcrypt unregistered [13124.268443] Key type ._llcrypt registered [13124.272409] Key type .llcrypt registered [13124.459375] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [13141.690781] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [13143.393585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13143.522650] alg: No test for adler32 (adler32-zlib) [13144.766222] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [13145.062610] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [13146.745135] Key type lgssc registered [13148.383277] Lustre: Echo OBD driver; http://www.lustre.org/ [13193.083976] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [13211.060254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13211.086332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.460591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13212.481677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13212.563389] Lustre: lustre-MDT0000: new disk, initializing [13212.718430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13212.749865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13218.301101] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13234.463590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13234.803407] Lustre: lustre-OST0000: new disk, initializing [13234.807979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13234.822491] Lustre: Skipped 1 previous similar message [13234.828472] Lustre: 437047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13234.922895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13242.858090] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13243.490451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13243.502864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13243.544396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13253.666409] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13270.087443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13270.145396] Lustre: 436029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13270.181728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13270.234642] Lustre: lustre-MDT0001: new disk, initializing [13270.398279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13270.464907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13270.486775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13274.856790] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13285.941235] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13286.307793] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13291.358187] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13291.752838] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13297.636074] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13297.640894] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13297.658857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13297.688436] Lustre: Skipped 1 previous similar message [13297.718260] Lustre: Skipped 1 previous similar message [13301.242539] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [13302.758067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13302.773042] Lustre: Skipped 1 previous similar message [13303.959569] Lustre: server umount lustre-MDT0001 complete [13312.995694] LustreError: 436040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13313.025797] LustreError: 436040:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [13318.137438] LustreError: 436039:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13318.158561] LustreError: 436039:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13323.236651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13323.240600] LustreError: 436040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13323.259240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13328.356037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13328.367178] LustreError: 436039:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13333.474107] LustreError: 436040:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13336.032152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13336.269691] Lustre: server umount lustre-OST0000 complete [13346.959913] Lustre: server umount lustre-MDT0000 complete [13362.855582] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [13366.486468] Key type lgssc unregistered [13366.871649] LNet: 440802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13366.888763] LNetError: 440802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13367.913870] LNet: Removed LNI 192.168.201.125@tcp [13368.987320] Key type .llcrypt unregistered [13368.993789] Key type ._llcrypt unregistered [13409.296856] Key type ._llcrypt registered [13409.303878] Key type .llcrypt registered [13409.428797] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [13409.744431] Key type .llcrypt unregistered [13409.746388] Key type ._llcrypt unregistered [13428.351875] Key type ._llcrypt registered [13428.353512] Key type .llcrypt registered [13428.471508] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [13429.573536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13429.600710] alg: No test for adler32 (adler32-zlib) [13430.764599] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [13430.994436] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [13432.664188] Key type lgssc registered [13433.916759] Lustre: Echo OBD driver; http://www.lustre.org/ [13460.446680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13460.457798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13461.695453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13461.751532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13461.827682] Lustre: lustre-MDT0000: new disk, initializing [13461.889186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13461.903588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13465.854547] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13475.947295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13476.075655] Lustre: 444185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13476.133800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13476.138588] Lustre: Skipped 1 previous similar message [13476.223525] Lustre: lustre-MDT0001: new disk, initializing [13476.375585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13476.406370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13476.410162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13481.451180] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13490.269860] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13496.020955] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13502.748899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13503.105388] Lustre: lustre-OST0000: new disk, initializing [13503.109675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13503.119610] Lustre: 446152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13503.222277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13509.744378] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13509.778794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13509.787570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13509.924814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13519.207341] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13527.522717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13527.541396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13527.570702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13530.605195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13530.607057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13532.156458] Lustre: server umount lustre-MDT0000 complete [13535.593763] LustreError: 444176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836923 with bad export cookie 11441877447781621529 [13535.596555] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13535.603273] LustreError: 444176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13535.718081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13535.740382] Lustre: Skipped 1 previous similar message [13535.757328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13535.763800] Lustre: Skipped 2 previous similar messages [13535.928133] Lustre: server umount lustre-MDT0001 complete [13542.221821] Lustre: server umount lustre-OST0000 complete [13553.446430] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:22:21 (1787836941) [13562.830330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13563.523466] LustreError: 448179:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13563.547743] LustreError: 448179:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13563.613704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13568.718295] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13569.003062] LustreError: 448180:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13573.094250] LustreError: 448179:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.219417] LustreError: 448180:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13580.201545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13580.708071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13585.854687] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13593.202534] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13598.416808] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13619.113839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13619.343655] Lustre: Found index 0 for lustre-OST0000, updating log [13619.538638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13625.922392] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13633.631307] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13633.911826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13639.140368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13639.142823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13639.160818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13639.177512] Lustre: Skipped 1 previous similar message [13640.692584] Lustre: server umount lustre-OST0000 complete [13656.446374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13656.447831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13656.454927] Lustre: Skipped 2 previous similar messages [13656.460732] Lustre: Skipped 2 previous similar messages [13662.916180] Lustre: server umount lustre-MDT0000 complete [13666.681786] LustreError: 448159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837054 with bad export cookie 11441877447781622922 [13666.695383] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13666.696244] LustreError: 448159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13667.132762] Lustre: server umount lustre-MDT0001 complete [13676.123683] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:24:23 (1787837063) [13680.928321] LNet: 1 local NIs in recovery (showing 1): 192.168.201.125@tcp [14294.933255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14295.573925] LustreError: 454592:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14295.665926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14301.164521] LustreError: 454593:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14301.654617] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14305.266548] LustreError: 454592:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14310.521122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14310.959797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14316.811329] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14326.561721] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14333.766358] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14346.528206] LNet: 1 local NIs in recovery (showing 1): 192.168.201.125@tcp [14354.283435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14354.640802] Lustre: Found index 0 for lustre-OST0000, updating log [14355.064432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14362.774550] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14371.021904] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14374.311615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14374.330197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14374.334225] Lustre: Skipped 3 previous similar messages [14379.553576] Lustre: server umount lustre-OST0000 complete [14383.489399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14383.492639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14383.519645] Lustre: Skipped 1 previous similar message [14383.541318] Lustre: Skipped 1 previous similar message [14385.570284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14385.593102] Lustre: Skipped 3 previous similar messages [14389.672106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14389.688280] Lustre: Skipped 7 previous similar messages [14390.144144] Lustre: server umount lustre-MDT0000 complete [14390.699082] LustreError: 454593:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14390.726211] LustreError: 454593:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [14394.423407] LustreError: 454572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837782 with bad export cookie 11441877447781624000 [14394.442977] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14394.452085] LustreError: 454572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14394.818654] Lustre: server umount lustre-MDT0001 complete [14405.920733] LNet: 1 local NIs in recovery (showing 1): 192.168.201.125@tcp [14428.298748] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [14438.099651] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [14478.151684] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:37:45 (1787837865) [14500.658295] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [14504.506910] Key type lgssc unregistered [14504.741465] LNet: 461211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14504.749469] LNetError: 461211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14616.360421] LNet: Removed LNI 192.168.201.125@tcp [14617.587044] Key type .llcrypt unregistered [14617.590547] Key type ._llcrypt unregistered [14638.541215] Key type ._llcrypt registered [14638.543515] Key type .llcrypt registered [14638.654876] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [14640.068745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14640.103704] alg: No test for adler32 (adler32-zlib) [14641.319080] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [14641.731401] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [14643.512467] Key type lgssc registered [14645.170410] Lustre: Echo OBD driver; http://www.lustre.org/ [14649.915940] LNet: Added LNI 192.168.201.125@tcp100 [8/256/0/180] [14650.967814] LNet: Added LNI 192.168.201.125@tcp101 [8/256/0/180] [14651.995051] LNet: Added LNI 192.168.201.125@tcp102 [8/256/0/180] [14653.021176] LNet: Added LNI 192.168.201.125@tcp103 [8/256/0/180] [14653.938909] LNet: Added LNI 192.168.201.125@tcp104 [8/256/0/180] [14654.841603] LNet: Added LNI 192.168.201.125@tcp105 [8/256/0/180] [14655.956872] LNet: Added LNI 192.168.201.125@tcp106 [8/256/0/180] [14656.968750] LNet: Added LNI 192.168.201.125@tcp107 [8/256/0/180] [14658.031335] LNet: Added LNI 192.168.201.125@tcp108 [8/256/0/180] [14658.865650] LNet: Added LNI 192.168.201.125@tcp109 [8/256/0/180] [14659.769154] LNet: Added LNI 192.168.201.125@tcp110 [8/256/0/180] [14660.792557] LNet: Added LNI 192.168.201.125@tcp111 [8/256/0/180] [14661.906395] LNet: Added LNI 192.168.201.125@tcp112 [8/256/0/180] [14662.750898] LNet: Added LNI 192.168.201.125@tcp113 [8/256/0/180] [14663.579108] LNet: Added LNI 192.168.201.125@tcp114 [8/256/0/180] [14664.509432] LNet: Added LNI 192.168.201.125@tcp115 [8/256/0/180] [14665.552955] LNet: Added LNI 192.168.201.125@tcp116 [8/256/0/180] [14666.717295] LNet: Added LNI 192.168.201.125@tcp117 [8/256/0/180] [14667.964189] LNet: Added LNI 192.168.201.125@tcp118 [8/256/0/180] [14669.045812] LNet: Added LNI 192.168.201.125@tcp119 [8/256/0/180] [14670.119947] LNet: Added LNI 192.168.201.125@tcp120 [8/256/0/180] [14671.250435] LNet: Added LNI 192.168.201.125@tcp121 [8/256/0/180] [14672.309329] LNet: Added LNI 192.168.201.125@tcp122 [8/256/0/180] [14673.585683] LNet: Added LNI 192.168.201.125@tcp123 [8/256/0/180] [14674.590760] LNet: Added LNI 192.168.201.125@tcp124 [8/256/0/180] [14675.569697] LNet: Added LNI 192.168.201.125@tcp125 [8/256/0/180] [14676.551248] LNet: Added LNI 192.168.201.125@tcp126 [8/256/0/180] [14677.568970] LNet: Added LNI 192.168.201.125@tcp127 [8/256/0/180] [14678.550116] LNet: Added LNI 192.168.201.125@tcp128 [8/256/0/180] [14679.571861] LNet: Added LNI 192.168.201.125@tcp129 [8/256/0/180] [14680.375934] LNet: Added LNI 192.168.201.125@tcp130 [8/256/0/180] [14681.228856] LNet: Added LNI 192.168.201.125@tcp131 [8/256/0/180] [14682.110885] LNet: Added LNI 192.168.201.125@tcp132 [8/256/0/180] [14682.927826] LNet: Added LNI 192.168.201.125@tcp133 [8/256/0/180] [14683.893577] LNet: Added LNI 192.168.201.125@tcp134 [8/256/0/180] [14684.958719] LNet: Added LNI 192.168.201.125@tcp135 [8/256/0/180] [14692.564957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14692.589438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14693.752251] Lustre: 464243:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.125@tcp: can't add NID 192.168.201.125@tcp130: rc = -75 [14693.897572] LNetError: 464268:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14693.927666] LNetError: Protocol error connecting to 192.168.201.125@tcp114 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14693.942174] LNetError: Skipped 4 previous similar messages [14693.984114] LNet: 7 local NIs in recovery (showing 7): 192.168.201.125@tcp124, 192.168.201.125@tcp125, 192.168.201.125@tcp126, 192.168.201.125@tcp128, 192.168.201.125@tcp129, 192.168.201.125@tcp127, 192.168.201.125@tcp130 [14694.019379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14694.048708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14694.121284] Lustre: lustre-MDT0000: new disk, initializing [14694.208109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14694.231740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14695.023641] LNetError: 464276:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14695.047632] LNetError: 464276:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14695.070512] LNetError: Protocol error connecting to 192.168.201.125@tcp128 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14695.095148] LNetError: Skipped 11 previous similar messages [14697.074377] LNetError: 464263:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14697.089766] LNetError: 464263:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14697.107065] LNetError: Protocol error connecting to 192.168.201.125@tcp128 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14699.687425] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14701.156603] LNetError: 461701:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14701.165525] LNetError: Protocol error connecting to 192.168.201.125@tcp128 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14709.349031] LNetError: 461702:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14709.366973] LNetError: Protocol error connecting to 192.168.201.125@tcp128 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14711.101384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14711.326798] Lustre: 464258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14711.412878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14711.416673] Lustre: Skipped 1 previous similar message [14711.553535] Lustre: lustre-MDT0001: new disk, initializing [14711.593539] Lustre: 465062:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.125@tcp: can't add NID 192.168.201.125@tcp130: rc = -75 [14711.610311] Lustre: 465062:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14711.750407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14711.781752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14711.797470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14717.203091] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14717.548439] Lustre: 465341:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.125@tcp: can't add NID 192.168.201.125@tcp130: rc = -75 [14717.561848] Lustre: 465341:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14718.565500] LNetError: 461703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14718.585372] LNetError: 461703:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14718.607447] LNetError: Protocol error connecting to 192.168.201.125@tcp134 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14718.623438] LNetError: Skipped 9 previous similar messages [14727.701691] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14734.428906] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14734.954415] LNetError: 464263:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14734.976820] LNetError: 464263:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14734.986677] LNetError: Protocol error connecting to 192.168.201.125@tcp129 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14735.002097] LNetError: Skipped 11 previous similar messages [14741.579101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14742.059427] Lustre: lustre-OST0000: new disk, initializing [14742.063613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14742.077869] Lustre: 466240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14742.200222] Lustre: 466245:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.125@tcp: can't add NID 192.168.201.125@tcp130: rc = -75 [14742.204930] Lustre: 466245:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14742.256205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14749.919628] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14752.317096] Lustre: 466697:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.125@tcp: can't add NID 192.168.201.125@tcp130: rc = -75 [14752.325445] Lustre: 466697:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14752.409157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14752.424146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14752.618739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14754.400660] LNet: 2 local NIs in recovery (showing 2): 192.168.201.125@tcp134, 192.168.201.125@tcp128 [14760.453623] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14768.098391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14768.104522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14768.105990] LustreError: Skipped 1 previous similar message [14768.116744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14768.128332] Lustre: Skipped 1 previous similar message [14768.738666] LNetError: 464274:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.125 [14768.761829] LNetError: 464274:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 25 previous similar messages [14768.769706] LNetError: Protocol error connecting to 192.168.201.125@tcp123 at host 192.168.201.125:988: is it running a compatible version of Lustre? [14768.795933] LNetError: Skipped 25 previous similar messages [14771.184401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14771.193639] Lustre: Skipped 1 previous similar message [14774.115615] Lustre: server umount lustre-OST0000 complete [14778.345952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14778.363754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14778.376758] Lustre: Skipped 1 previous similar message [14778.388216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14781.409705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14781.419697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14784.495850] Lustre: server umount lustre-MDT0000 complete [14788.913989] LustreError: 464251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838177 with bad export cookie 15953581566653039434 [14788.922456] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14788.927702] LustreError: 464251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14789.400884] Lustre: server umount lustre-MDT0001 complete [14803.143969] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [14807.160408] Key type lgssc unregistered [14807.842415] LNet: 468156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14807.857708] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14807.865364] LNet: Removed LNI 192.168.201.125@tcp [14808.928175] LNet: Removed LNI 192.168.201.125@tcp100 [14808.951296] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14808.971488] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14808.991324] LNet: Removed LNI 192.168.201.125@tcp101 [14810.019788] LNet: Removed LNI 192.168.201.125@tcp102 [14810.057703] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14810.083379] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14810.097322] LNet: Removed LNI 192.168.201.125@tcp103 [14811.105083] LNet: Removed LNI 192.168.201.125@tcp104 [14812.128571] LNet: Removed LNI 192.168.201.125@tcp105 [14812.139975] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14812.176677] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14812.197393] LNet: Removed LNI 192.168.201.125@tcp106 [14813.216552] LNet: Removed LNI 192.168.201.125@tcp107 [14814.240425] LNet: Removed LNI 192.168.201.125@tcp108 [14815.264507] LNet: Removed LNI 192.168.201.125@tcp109 [14816.288171] LNet: Removed LNI 192.168.201.125@tcp110 [14816.291456] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14816.315304] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14816.323275] LNet: Removed LNI 192.168.201.125@tcp111 [14817.378672] LNet: Removed LNI 192.168.201.125@tcp112 [14818.401296] LNet: Removed LNI 192.168.201.125@tcp113 [14819.424550] LNet: Removed LNI 192.168.201.125@tcp114 [14820.448583] LNet: Removed LNI 192.168.201.125@tcp115 [14821.472217] LNet: Removed LNI 192.168.201.125@tcp116 [14822.496565] LNet: Removed LNI 192.168.201.125@tcp117 [14823.520173] LNet: Removed LNI 192.168.201.125@tcp118 [14824.544136] LNet: Removed LNI 192.168.201.125@tcp119 [14824.551504] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14824.568917] LNetError: 468156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14824.589339] LNet: Removed LNI 192.168.201.125@tcp120 [14825.632324] LNet: Removed LNI 192.168.201.125@tcp121 [14826.656342] LNet: Removed LNI 192.168.201.125@tcp122 [14827.680458] LNet: Removed LNI 192.168.201.125@tcp123 [14828.704267] LNet: Removed LNI 192.168.201.125@tcp124 [14829.728241] LNet: Removed LNI 192.168.201.125@tcp125 [14830.752991] LNet: Removed LNI 192.168.201.125@tcp126 [14831.776447] LNet: Removed LNI 192.168.201.125@tcp127 [14831.780628] LNet: Removed LNI 192.168.201.125@tcp128 [14832.801109] LNet: Removed LNI 192.168.201.125@tcp129 [14833.824178] LNet: Removed LNI 192.168.201.125@tcp130 [14834.855059] LNet: Removed LNI 192.168.201.125@tcp131 [14835.873547] LNet: Removed LNI 192.168.201.125@tcp132 [14836.896280] LNet: Removed LNI 192.168.201.125@tcp133 [14836.898321] LNet: Removed LNI 192.168.201.125@tcp134 [14837.934449] LNet: Removed LNI 192.168.201.125@tcp135 [14838.938858] Key type .llcrypt unregistered [14838.941230] Key type ._llcrypt unregistered [14864.694680] Key type ._llcrypt registered [14864.697831] Key type .llcrypt registered [14864.838525] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [14881.258866] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [14882.835361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14882.870766] alg: No test for adler32 (adler32-zlib) [14884.140105] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [14884.532137] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [14886.297486] Key type lgssc registered [14887.584529] Lustre: Echo OBD driver; http://www.lustre.org/ [14925.110361] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:45:12 (1787838312) [14945.200512] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [14948.186575] Key type lgssc unregistered [14948.472719] LNet: 472251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14948.479848] LNetError: 472251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14948.503242] LNet: Removed LNI 192.168.201.125@tcp [14949.388952] Key type .llcrypt unregistered [14949.391447] Key type ._llcrypt unregistered [14969.813201] Key type ._llcrypt registered [14969.816714] Key type .llcrypt registered [14969.967157] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [14971.286857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14971.544664] alg: No test for adler32 (adler32-zlib) [14972.800913] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [14973.034119] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [14974.712265] Key type lgssc registered [14976.271506] Lustre: Echo OBD driver; http://www.lustre.org/ [14987.481989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14987.519461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14988.919310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14988.966168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14989.061227] Lustre: lustre-MDT0000: new disk, initializing [14989.163692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14989.184216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14995.157616] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15007.112791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15007.274824] Lustre: 473466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15007.338147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15007.348685] Lustre: Skipped 1 previous similar message [15007.468429] Lustre: lustre-MDT0001: new disk, initializing [15007.644625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15007.721566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15007.737367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15013.075672] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15023.027935] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15029.113480] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15036.000675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15036.349787] Lustre: lustre-OST0000: new disk, initializing [15036.353245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15036.367478] Lustre: 475435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15036.493529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15043.317808] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15043.633721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15043.637907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15043.723569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15054.647579] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15059.559183] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing lnet_if_list [15066.266667] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15066.306345] LNet: Added LNI 192.168.201.125@tcp10 [8/256/0/180] [15072.180720] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15072.210835] LNet: Added LNI 192.168.201.125@tcp11 [8/256/0/180] [15077.345942] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15077.384943] LNet: Added LNI 192.168.201.125@tcp12 [8/256/0/180] [15083.293215] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15083.329528] LNet: Added LNI 192.168.201.125@tcp13 [8/256/0/180] [15089.646422] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15089.683273] LNet: Added LNI 192.168.201.125@tcp14 [8/256/0/180] [15094.627987] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15094.651490] LNet: Added LNI 192.168.201.125@tcp15 [8/256/0/180] [15100.647780] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15100.704862] LNet: Added LNI 192.168.201.125@tcp16 [8/256/0/180] [15106.877498] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15106.916572] LNet: Added LNI 192.168.201.125@tcp17 [8/256/0/180] [15113.043557] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15113.074139] LNet: Added LNI 192.168.201.125@tcp18 [8/256/0/180] [15118.965566] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15119.011457] LNet: Added LNI 192.168.201.125@tcp19 [8/256/0/180] [15126.245425] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15126.304739] LNet: Added LNI 192.168.201.125@tcp20 [8/256/0/180] [15131.239813] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15131.255888] LNet: Added LNI 192.168.201.125@tcp21 [8/256/0/180] [15138.547494] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15138.598557] LNet: Added LNI 192.168.201.125@tcp22 [8/256/0/180] [15144.085887] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15144.121837] LNet: Added LNI 192.168.201.125@tcp23 [8/256/0/180] [15150.546690] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15150.567905] LNet: Added LNI 192.168.201.125@tcp24 [8/256/0/180] [15157.212131] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15157.231803] LNet: Added LNI 192.168.201.125@tcp25 [8/256/0/180] [15162.181759] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15162.239717] LNet: Added LNI 192.168.201.125@tcp26 [8/256/0/180] [15169.385295] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15169.431893] LNet: Added LNI 192.168.201.125@tcp27 [8/256/0/180] [15176.526564] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15176.567836] LNet: Added LNI 192.168.201.125@tcp28 [8/256/0/180] [15184.240188] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15184.258184] LNet: Added LNI 192.168.201.125@tcp29 [8/256/0/180] [15190.025813] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15190.060636] LNet: Added LNI 192.168.201.125@tcp30 [8/256/0/180] [15197.693373] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15197.722025] LNet: Added LNI 192.168.201.125@tcp31 [8/256/0/180] [15206.777233] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15206.834935] LNet: Added LNI 192.168.201.125@tcp32 [8/256/0/180] [15215.985246] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15216.010113] LNet: Added LNI 192.168.201.125@tcp33 [8/256/0/180] [15224.188716] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15224.221455] LNet: Added LNI 192.168.201.125@tcp34 [8/256/0/180] [15231.401711] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15231.438395] LNet: Added LNI 192.168.201.125@tcp35 [8/256/0/180] [15239.451404] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15239.477045] LNet: Added LNI 192.168.201.125@tcp36 [8/256/0/180] [15247.366683] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15247.400699] LNet: Added LNI 192.168.201.125@tcp37 [8/256/0/180] [15254.960517] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15255.006129] LNet: Added LNI 192.168.201.125@tcp38 [8/256/0/180] [15262.959398] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15262.991327] LNet: Added LNI 192.168.201.125@tcp39 [8/256/0/180] [15270.598384] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15270.619664] LNet: Added LNI 192.168.201.125@tcp40 [8/256/0/180] [15278.859205] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15278.893090] LNet: Added LNI 192.168.201.125@tcp41 [8/256/0/180] [15286.329918] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15286.383985] LNet: Added LNI 192.168.201.125@tcp42 [8/256/0/180] [15293.456214] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15293.488766] LNet: Added LNI 192.168.201.125@tcp43 [8/256/0/180] [15301.690357] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15301.719679] LNet: Added LNI 192.168.201.125@tcp44 [8/256/0/180] [15309.875439] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15309.924520] LNet: Added LNI 192.168.201.125@tcp45 [8/256/0/180] [15319.794880] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15319.818681] LNet: Added LNI 192.168.201.125@tcp46 [8/256/0/180] [15328.580908] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15328.601263] LNet: Added LNI 192.168.201.125@tcp47 [8/256/0/180] [15336.531025] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15336.579650] LNet: Added LNI 192.168.201.125@tcp48 [8/256/0/180] [15345.855709] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15345.890891] LNet: Added LNI 192.168.201.125@tcp49 [8/256/0/180] [15355.082681] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15355.107270] LNet: Added LNI 192.168.201.125@tcp50 [8/256/0/180] [15376.356448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15376.376923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15376.403683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15379.660697] Lustre: server umount lustre-MDT0000 complete [15383.238258] LustreError: 475434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838771 with bad export cookie 18022319649355518798 [15383.245726] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15383.248219] LustreError: 475434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15383.575439] Lustre: server umount lustre-MDT0001 complete [15401.513955] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [15403.489195] Lustre: 472738:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838775/real 1787838775] req@00000000a8186c0a x1874684397398656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15403.509110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15403.520386] Lustre: Skipped 2 previous similar messages [15404.514778] Lustre: 472739:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838776/real 1787838776] req@00000000e2b8ac91 x1874684397398784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15408.610180] Lustre: 472736:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838780/real 1787838780] req@00000000015ef5f2 x1874684397398912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15410.044492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15414.753882] Lustre: 472738:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838786/real 1787838786] req@00000000d6b93f79 x1874684397399296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15414.782764] Lustre: 472738:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15418.848540] Lustre: 472739:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787838791/real 1787838791] req@00000000898b7a42 x1874684397399424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787838807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15428.131455] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfa1c243c1860ddc7 [15428.152451] Lustre: MGC192.168.201.125@tcp: Connection restored to 0@lo (at 0@lo) [15428.416830] LustreError: 475426:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15428.443711] LustreError: 475426:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [15428.483485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15429.478972] LustreError: 475427:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15429.502622] LustreError: 475427:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15432.548169] LustreError: 499683:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15432.555545] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15432.570733] LustreError: 499683:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15433.990972] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15437.802297] LustreError: 499681:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15437.818482] LustreError: 499681:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15442.923253] LustreError: 475427:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15442.949456] LustreError: 475427:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15443.387155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15444.010796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15449.086136] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15449.774191] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15457.841093] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15463.871989] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15484.899480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15484.902486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15484.914508] LustreError: Skipped 1 previous similar message [15484.920142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15484.935587] Lustre: Skipped 1 previous similar message [15484.958127] Lustre: Skipped 3 previous similar messages [15489.538520] Lustre: server umount lustre-OST0000 complete [15495.138096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15495.140127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15495.147091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15495.174953] Lustre: Skipped 1 previous similar message [15499.909374] Lustre: server umount lustre-MDT0000 complete [15500.257583] LustreError: 499688:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15500.283483] LustreError: 499688:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15503.902330] LustreError: 473459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838892 with bad export cookie 18022319649355521479 [15503.906229] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15503.925480] LustreError: 473459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15504.448075] Lustre: server umount lustre-MDT0001 complete [15516.643549] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [15519.354750] Key type lgssc unregistered [15519.748394] LNet: 502489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15519.757464] LNetError: 502489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15519.765415] LNet: Removed LNI 192.168.201.125@tcp [15519.767037] LNet: Removed LNI 192.168.201.125@tcp10 [15519.768800] LNet: Removed LNI 192.168.201.125@tcp11 [15519.773843] LNet: Removed LNI 192.168.201.125@tcp12 [15519.779375] LNet: Removed LNI 192.168.201.125@tcp13 [15519.781241] LNet: Removed LNI 192.168.201.125@tcp14 [15519.782981] LNet: Removed LNI 192.168.201.125@tcp15 [15519.785601] LNet: Removed LNI 192.168.201.125@tcp16 [15519.788534] LNet: Removed LNI 192.168.201.125@tcp17 [15519.791414] LNet: Removed LNI 192.168.201.125@tcp18 [15519.800381] LNet: Removed LNI 192.168.201.125@tcp19 [15519.806736] LNet: Removed LNI 192.168.201.125@tcp20 [15519.811964] LNet: Removed LNI 192.168.201.125@tcp21 [15519.813884] LNet: Removed LNI 192.168.201.125@tcp22 [15519.821514] LNet: Removed LNI 192.168.201.125@tcp23 [15519.827856] LNet: Removed LNI 192.168.201.125@tcp24 [15519.838302] LNet: Removed LNI 192.168.201.125@tcp25 [15519.840251] LNet: Removed LNI 192.168.201.125@tcp26 [15519.846423] LNet: Removed LNI 192.168.201.125@tcp27 [15519.853801] LNet: Removed LNI 192.168.201.125@tcp28 [15519.857974] LNet: Removed LNI 192.168.201.125@tcp29 [15519.863278] LNet: Removed LNI 192.168.201.125@tcp30 [15519.868570] LNet: Removed LNI 192.168.201.125@tcp31 [15519.875439] LNet: Removed LNI 192.168.201.125@tcp32 [15519.880796] LNet: Removed LNI 192.168.201.125@tcp33 [15519.885925] LNet: Removed LNI 192.168.201.125@tcp34 [15519.887628] LNet: Removed LNI 192.168.201.125@tcp35 [15519.897243] LNet: Removed LNI 192.168.201.125@tcp36 [15519.899184] LNet: Removed LNI 192.168.201.125@tcp37 [15519.908555] LNet: Removed LNI 192.168.201.125@tcp38 [15519.910542] LNet: Removed LNI 192.168.201.125@tcp39 [15519.916711] LNet: Removed LNI 192.168.201.125@tcp40 [15519.919812] LNet: Removed LNI 192.168.201.125@tcp41 [15519.930288] LNet: Removed LNI 192.168.201.125@tcp42 [15519.939840] LNet: Removed LNI 192.168.201.125@tcp43 [15519.949232] LNet: Removed LNI 192.168.201.125@tcp44 [15519.957517] LNet: Removed LNI 192.168.201.125@tcp45 [15519.960583] LNet: Removed LNI 192.168.201.125@tcp46 [15519.967216] LNet: Removed LNI 192.168.201.125@tcp47 [15519.983901] LNet: Removed LNI 192.168.201.125@tcp48 [15519.989140] LNet: Removed LNI 192.168.201.125@tcp49 [15520.001078] LNet: Removed LNI 192.168.201.125@tcp50 [15520.749509] Key type .llcrypt unregistered [15520.751579] Key type ._llcrypt unregistered [15543.056460] Key type ._llcrypt registered [15543.059257] Key type .llcrypt registered [15543.197675] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [15560.203099] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [15561.888861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15561.941714] alg: No test for adler32 (adler32-zlib) [15563.209194] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [15563.633524] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [15565.456911] Key type lgssc registered [15567.129391] Lustre: Echo OBD driver; http://www.lustre.org/ [15610.802312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15610.842331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15612.229142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15612.288318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15612.424427] Lustre: lustre-MDT0000: new disk, initializing [15612.589350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15612.639238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15618.756783] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15630.644494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15630.809169] Lustre: 505689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15630.865628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15630.872792] Lustre: Skipped 1 previous similar message [15630.986583] Lustre: lustre-MDT0001: new disk, initializing [15631.081506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15631.138755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15631.147471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15636.135770] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15648.876680] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15654.997554] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15663.846523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15664.118119] Lustre: *** cfs_fail_loc=90f, val=0*** [15664.231557] Lustre: lustre-OST0000: new disk, initializing [15664.234759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15664.240499] Lustre: 507706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15664.342769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15672.507795] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15673.438798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15673.447972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15673.561692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15683.455367] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15694.306404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15694.315670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15694.327737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15694.334817] Lustre: Skipped 1 previous similar message [15699.429285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15699.446729] Lustre: Skipped 1 previous similar message [15700.638777] Lustre: server umount lustre-OST0000 complete [15708.135816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15708.151477] LustreError: Skipped 1 previous similar message [15708.163292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15708.187758] Lustre: Skipped 1 previous similar message [15708.198675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15709.665652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15712.619399] Lustre: server umount lustre-MDT0000 complete [15717.218159] LustreError: 505679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839105 with bad export cookie 1420996031605465796 [15717.229689] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15717.236432] LustreError: 505679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15717.988612] Lustre: server umount lustre-MDT0001 complete [15732.121166] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [15735.376819] Key type lgssc unregistered [15735.737990] LNet: 509617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15735.745853] LNetError: 509617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15736.820107] LNet: Removed LNI 192.168.201.125@tcp [15737.896943] Key type .llcrypt unregistered [15737.899565] Key type ._llcrypt unregistered [15759.560595] Key type ._llcrypt registered [15759.566251] Key type .llcrypt registered [15759.716855] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [15761.601552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15761.644074] alg: No test for adler32 (adler32-zlib) [15762.852880] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [15763.291763] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [15765.128191] Key type lgssc registered [15766.418570] Lustre: Echo OBD driver; http://www.lustre.org/ [15775.086856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15775.108613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15776.768564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15781.884049] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15785.960206] LustreError: 510892:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15785.992413] LustreError: 510892:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [15791.074830] LustreError: 510891:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15791.382583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15791.987717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15797.354458] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15806.831690] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15813.753589] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15821.221938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15821.636444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15827.702631] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15835.767392] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15848.011248] loop0: detected capacity change from 0 to 204800000 [15848.229407] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15848.245950] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15848.285861] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15855.322640] loop0: detected capacity change from 0 to 204800000 [15855.544408] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15855.578485] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15855.615759] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15861.875251] loop0: detected capacity change from 0 to 204800000 [15867.574753] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15867.704693] Lustre: fs15246-OST0000: new disk, initializing [15867.708737] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15867.714432] Lustre: 514486:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15867.780034] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15874.419886] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15884.121945] loop1: detected capacity change from 0 to 204800000 [15890.409846] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15890.584786] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15890.659773] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15890.823395] Lustre: fs15246-MDT0000: new disk, initializing [15891.084452] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15891.138570] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15891.276861] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15892.334659] Lustre: 515618:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b2f0adaf x1874685225649152/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787839296 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15895.871206] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [15897.510686] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15897.537437] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15897.577110] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15917.543920] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15917.585933] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15921.735772] Lustre: server umount fs15246-MDT0000 complete [15922.662719] LustreError: 510888:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15927.780391] LustreError: 510886:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15931.530179] Lustre: server umount fs15246-OST0000 complete [15943.146066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15943.163070] Lustre: Skipped 1 previous similar message [15943.163919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15943.172400] Lustre: Skipped 1 previous similar message [15948.257885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15948.288841] Lustre: Skipped 1 previous similar message [15953.380858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15953.389672] Lustre: Skipped 1 previous similar message [15956.960252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15957.275813] Lustre: server umount lustre-OST0000 complete [15963.617905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15963.620141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15963.637336] Lustre: Skipped 1 previous similar message [15963.663758] Lustre: Skipped 1 previous similar message [15967.550201] Lustre: server umount lustre-MDT0000 complete [15968.748144] LustreError: 512803:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15968.762886] LustreError: 512803:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15971.865919] LustreError: 510870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839360 with bad export cookie 218554343347741145 [15971.869356] LustreError: 510870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15971.872044] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15972.083348] Lustre: server umount lustre-MDT0001 complete [15987.498360] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [15991.910608] Key type lgssc unregistered [15992.266586] LNet: 518733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15992.271179] LNetError: 518733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15992.286890] LNet: Removed LNI 192.168.201.125@tcp [15993.594842] Key type .llcrypt unregistered [15993.596891] Key type ._llcrypt unregistered [16016.472429] Key type ._llcrypt registered [16016.473943] Key type .llcrypt registered [16016.586604] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [16017.849950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16017.893070] alg: No test for adler32 (adler32-zlib) [16018.891387] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [16019.087345] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [16020.760408] Key type lgssc registered [16021.774936] Lustre: Echo OBD driver; http://www.lustre.org/ [16060.058924] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [16072.580175] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [16110.657340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16110.683832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16112.087650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16112.131117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16112.293595] Lustre: lustre-MDT0000: new disk, initializing [16112.463831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16112.482313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16118.068726] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [16130.571523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16130.680288] Lustre: 523200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16130.735398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16130.740849] Lustre: Skipped 1 previous similar message [16130.835367] Lustre: lustre-MDT0001: new disk, initializing [16130.947000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16130.980975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16130.994083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16137.195678] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [16147.634744] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16153.957427] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16161.794373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16162.149333] Lustre: lustre-OST0000: new disk, initializing [16162.152750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16162.162613] Lustre: 525171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16162.271884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16169.617639] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [16172.133967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16172.141248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16172.290258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16181.013573] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16186.619260] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16186.827930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16192.425713] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16192.692257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16198.115419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16198.118220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16198.122786] LustreError: Skipped 1 previous similar message [16198.140065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16198.142875] Lustre: Skipped 1 previous similar message [16198.157048] Lustre: Skipped 1 previous similar message [16200.969558] Lustre: server umount lustre-OST0000 complete [16207.845194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16207.861088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16207.892705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16212.533832] Lustre: server umount lustre-MDT0000 complete [16217.131661] LustreError: 523192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839605 with bad export cookie 1441656586474665208 [16217.133357] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16217.156863] LustreError: 523192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16217.684353] Lustre: server umount lustre-MDT0001 complete [16250.619130] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [16267.473577] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [16301.798246] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15996 sec ========= 10:08:09 (1787839689) [16303.766537] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:08:11 (1787839691) === [16307.552631] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:08:14 (1787839694) === [16353.126235] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [16356.619170] Key type lgssc unregistered [16356.958421] LNet: 531841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16356.965422] LNetError: 531841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16357.012068] LNet: Removed LNI 192.168.201.125@tcp [16358.302399] Key type .llcrypt unregistered [16358.308108] Key type ._llcrypt unregistered