[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507537303 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003076] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008266] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011007] pid_max: default: 32768 minimum: 301 [ 0.012141] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014028] SELinux: Initializing. [ 0.015070] *** VALIDATE selinux *** [ 0.024255] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029263] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] *** VALIDATE tmpfs *** [ 0.034246] *** VALIDATE proc *** [ 0.035208] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.038173] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041026] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045067] debug: unmapping init [mem 0xffffffffa7459000-0xffffffffa7460fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048661] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056304] rcu: Hierarchical SRCU implementation. [ 0.058570] smp: Bringing up secondary CPUs ... [ 0.059584] x86: Booting SMP configuration: [ 0.060023] .... node #0, CPUs: #1 #2 #3 [ 0.066665] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128369] node 0 deferred pages initialised in 57ms [ 0.131124] devtmpfs: initialized [ 0.132271] x86/mm: Memory block size: 128MB [ 0.134280] gcov: version magic: 0x41383552 [ 0.136378] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137190] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138205] pinctrl core: initialized pinctrl subsystem [ 0.139124] [ 0.139421] ************************************************************* [ 0.140012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141013] ** ** [ 0.142012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143011] ** ** [ 0.144014] ** This means that this kernel is built to expose internal ** [ 0.145012] ** IOMMU data structures, which may compromise security on ** [ 0.146015] ** your system. ** [ 0.147011] ** ** [ 0.148014] ** If you see this message and you are not debugging the ** [ 0.149015] ** kernel, report this immediately to your vendor! ** [ 0.150013] ** ** [ 0.151013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ************************************************************* [ 0.153772] NET: Registered protocol family 16 [ 0.154448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157349] cpuidle: using governor menu [ 0.158000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.161301] PCI: Using configuration type 1 for base access [ 0.162135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170082] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.173076] cryptd: max_cpu_qlen set to 1000 [ 0.174216] ACPI: Added _OSI(Module Device) [ 0.175015] ACPI: Added _OSI(Processor Device) [ 0.176011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.177008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.180425] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182572] ACPI: Interpreter enabled [ 0.183044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183881] ACPI: Using IOAPIC for interrupt routing [ 0.184090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.185328] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.192482] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.193030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.194015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.195064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.197100] acpiphp: Slot [2] registered [ 0.198147] acpiphp: Slot [5] registered [ 0.199063] acpiphp: Slot [6] registered [ 0.200047] acpiphp: Slot [7] registered [ 0.201064] acpiphp: Slot [8] registered [ 0.202108] acpiphp: Slot [9] registered [ 0.203070] acpiphp: Slot [10] registered [ 0.204148] acpiphp: Slot [3] registered [ 0.205076] acpiphp: Slot [4] registered [ 0.205962] acpiphp: Slot [11] registered [ 0.206078] acpiphp: Slot [12] registered [ 0.206993] acpiphp: Slot [13] registered [ 0.207077] acpiphp: Slot [14] registered [ 0.208083] acpiphp: Slot [15] registered [ 0.209077] acpiphp: Slot [16] registered [ 0.210053] acpiphp: Slot [17] registered [ 0.211024] acpiphp: Slot [18] registered [ 0.212054] acpiphp: Slot [19] registered [ 0.213042] acpiphp: Slot [20] registered [ 0.214061] acpiphp: Slot [21] registered [ 0.215084] acpiphp: Slot [22] registered [ 0.216058] acpiphp: Slot [23] registered [ 0.217023] acpiphp: Slot [24] registered [ 0.217972] acpiphp: Slot [25] registered [ 0.218069] acpiphp: Slot [26] registered [ 0.219056] acpiphp: Slot [27] registered [ 0.220079] acpiphp: Slot [28] registered [ 0.221059] acpiphp: Slot [29] registered [ 0.222055] acpiphp: Slot [30] registered [ 0.223085] acpiphp: Slot [31] registered [ 0.224039] PCI host bridge to bus 0000:00 [ 0.225016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.226015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.227017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.228014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.229017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.230016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.231119] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.232659] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.233865] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.237874] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.239515] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.240015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.241010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.242016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.243479] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.246535] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.248039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.251973] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.258016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.271016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.279015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.286317] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.299027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.309026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.331025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.340168] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.352016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.358013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.378016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.386825] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.393024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.399021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.413025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.423000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.429023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.434023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.448021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.462012] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.468016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.475015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.491014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.500315] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.506018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.516024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.529026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.540405] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.542327] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.544304] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.546285] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.548193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.552032] iommu: Default domain type: Passthrough [ 0.553286] SCSI subsystem initialized [ 0.554088] ACPI: bus type USB registered [ 0.555067] usbcore: registered new interface driver usbfs [ 0.556070] usbcore: registered new interface driver hub [ 0.557053] usbcore: registered new device driver usb [ 0.558110] pps_core: LinuxPPS API ver. 1 registered [ 0.560006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.562044] PTP clock support registered [ 0.563151] EDAC MC: Ver: 3.0.0 [ 0.565130] PCI: Using ACPI for IRQ routing [ 0.567095] NetLabel: Initializing [ 0.569015] NetLabel: domain hash size = 128 [ 0.571016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.573103] NetLabel: unlabeled traffic allowed by default [ 0.575185] vgaarb: loaded [ 0.577237] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.578008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.587202] clocksource: Switched to clocksource kvm-clock [ 0.671762] VFS: Disk quotas dquot_6.6.0 [ 0.672728] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.674257] *** VALIDATE ramfs *** [ 0.675074] *** VALIDATE hugetlbfs *** [ 0.676424] pnp: PnP ACPI init [ 0.677889] pnp: PnP ACPI: found 6 devices [ 0.692282] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.694410] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.695800] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.697259] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.698776] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.700154] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.701876] NET: Registered protocol family 2 [ 0.704023] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.708455] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.712686] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.720536] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.724853] TCP: Hash tables configured (established 65536 bind 65536) [ 0.727724] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.730228] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.732245] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.735173] NET: Registered protocol family 1 [ 0.736980] RPC: Registered named UNIX socket transport module. [ 0.738444] RPC: Registered udp transport module. [ 0.739676] RPC: Registered tcp transport module. [ 0.740701] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.742575] NET: Registered protocol family 44 [ 0.743991] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.745545] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.747166] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.749182] PCI: CLS 0 bytes, default 64 [ 0.750432] Unpacking initramfs... [ 2.263893] debug: unmapping init [mem 0xffff94e27cc54000-0xffff94e27ffbffff] [ 2.267319] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.269391] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.272496] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.065164] Initialise system trusted keyrings [ 3.067851] Key type blacklist registered [ 3.081051] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.088988] zbud: loaded [ 3.092433] *** VALIDATE nfs *** [ 3.093532] *** VALIDATE nfs4 *** [ 3.095080] pstore: using deflate compression [ 3.098853] Platform Keyring initialized [ 3.227884] NET: Registered protocol family 38 [ 3.231828] Key type asymmetric registered [ 3.234438] Asymmetric key parser 'x509' registered [ 3.236842] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.241932] io scheduler mq-deadline registered [ 3.243904] io scheduler kyber registered [ 3.245959] io scheduler bfq registered [ 3.247977] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.252413] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.256520] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.260317] ACPI: Power Button [PWRF] [ 3.265495] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.271430] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.280767] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.286910] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.301214] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.329637] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.360603] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.365780] Non-volatile memory driver v1.3 [ 3.367986] Linux agpgart interface v0.103 [ 3.405257] virtio_blk virtio1: [vda] 134592 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.408589] vda: detected capacity change from 0 to 68911104 [ 3.424595] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.427790] vdb: detected capacity change from 0 to 1073741824 [ 3.446947] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.449448] vdc: detected capacity change from 0 to 2621440000 [ 3.460459] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.462255] vdd: detected capacity change from 0 to 2621440000 [ 3.481677] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.485501] vde: detected capacity change from 0 to 4294967296 [ 3.521523] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.525371] vdf: detected capacity change from 0 to 4294967296 [ 3.539536] libphy: Fixed MDIO Bus: probed [ 3.556623] usbcore: registered new interface driver usbserial_generic [ 3.558861] usbserial: USB Serial support registered for generic [ 3.560773] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.567870] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.569655] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.572420] mousedev: PS/2 mouse device common for all mice [ 3.575573] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.581321] rtc_cmos 00:05: RTC can wake from S4 [ 3.583657] rtc_cmos 00:05: registered as rtc0 [ 3.585940] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.590556] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.601066] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.601986] intel_pstate: CPU model not supported [ 3.607429] hid: raw HID events driver (C) Jiri Kosina [ 3.612224] usbcore: registered new interface driver usbhid [ 3.613814] usbhid: USB HID core driver [ 3.615414] drop_monitor: Initializing network drop monitor service [ 3.617451] Initializing XFRM netlink socket [ 3.619235] NET: Registered protocol family 10 [ 3.622130] Segment Routing with IPv6 [ 3.623322] NET: Registered protocol family 17 [ 3.625639] mpls_gso: MPLS GSO support [ 3.632492] RAS: Correctable Errors collector initialized. [ 3.634538] AVX version of gcm_enc/dec engaged. [ 3.636000] AES CTR mode by8 optimization enabled [ 3.813814] sched_clock: Marking stable (3813792124, 0)->(4744579479, -930787355) [ 3.819195] registered taskstats version 1 [ 3.822196] Loading compiled-in X.509 certificates [ 3.824384] zswap: loaded using pool lzo/zbud [ 3.874741] Key type big_key registered [ 3.912786] Key type encrypted registered [ 3.916560] ima: No TPM chip found, activating TPM-bypass! [ 3.925649] ima: Allocated hash algorithm: sha1 [ 3.928779] ima: No architecture policies found [ 3.932287] evm: Initialising EVM extended attributes: [ 3.935228] evm: security.selinux [ 3.936724] evm: security.ima [ 3.938173] evm: security.capability [ 3.941174] evm: HMAC attrs: 0x1 [ 3.943783] rtc_cmos 00:05: setting system clock to 2026-03-15 20:31:16 UTC (1773606676) [ 3.953376] debug: unmapping init [mem 0xffffffffa8403000-0xffffffffa85fffff] [ 3.961262] debug: unmapping init [mem 0xffffffffa7182000-0xffffffffa7458fff] [ 3.974046] Write protecting the kernel read-only data: 28672k [ 3.986965] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 3.994578] debug: unmapping init [mem 0xffffffffa6114000-0xffffffffa61fffff] [ 4.078715] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.094701] systemd[1]: Detected virtualization kvm. [ 4.100576] systemd[1]: Detected architecture x86-64. [ 4.102722] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.138266] systemd[1]: No hostname configured. [ 4.142957] systemd[1]: Set hostname to . [ 4.145671] random: systemd: uninitialized urandom read (16 bytes read) [ 4.148826] systemd[1]: Initializing machine ID from random generator. [ 4.276667] random: ln: uninitialized urandom read (6 bytes read) [ 4.457739] random: systemd: uninitialized urandom read (16 bytes read) [ 4.460794] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.467735] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.480773] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.369822] device-mapper: uevent: version 1.0.3 [ 6.372590] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.219666] virtio_net virtio0 ens2: renamed from eth0 [ 8.383165] random: fast init done [ 8.881986] scsi host0: ata_piix [ 9.064053] scsi host1: ata_piix [ 9.074389] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.085927] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.459639] random: crng init done [ 13.463415] random: 7 urandom warning(s) missed due to ratelimiting [ 15.940927] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.159588] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.652069] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.120231] SELinux: Disabled at runtime. [ 22.229450] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.258426] systemd[1]: Detected virtualization kvm. [ 22.266746] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.734973] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.743125] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.761141] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.773814] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.780743] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.795363] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.810632] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ 25.212905] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.557442] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.761149] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.799726] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.479037] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.558864] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 33.028652] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 33.770724] NFS: Registering the id_resolver key type [ 33.774380] Key type id_resolver registered [ 33.776773] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg334-server login: [ 53.583378] hrtimer: interrupt took 13334221 ns [ 101.604048] libcfs: loading out-of-tree module taints kernel. [ 101.687324] Key type ._llcrypt registered [ 101.689253] Key type .llcrypt registered [ 101.870859] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 121.157978] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 123.292714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.307217] alg: No test for adler32 (adler32-zlib) [ 124.870947] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 125.717801] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 127.440557] Key type lgssc registered [ 129.785608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.364557] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.291991] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 220.607702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 220.670139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.150701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 222.191793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 222.375885] Lustre: lustre-MDT0000: new disk, initializing [ 222.567082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.626608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.995641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.157386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.350801] Lustre: 6514:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 241.443072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 241.447585] Lustre: Skipped 1 previous similar message [ 241.724101] Lustre: lustre-MDT0001: new disk, initializing [ 241.821329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.862461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.881421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.770138] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.754898] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.993836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.844853] Lustre: lustre-OST0000: new disk, initializing [ 261.862784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.993293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.075691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.096643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.203299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 268.558160] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.404880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.553565] Lustre: lustre-OST0001: new disk, initializing [ 283.556887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 283.629374] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 290.053402] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.454386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 293.465517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 293.552554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 302.731510] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 308.093858] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.135803] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.644655] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing yml_node [ 328.730345] Lustre: DEBUG MARKER: Client: 2.17.51.2 [ 331.146894] Lustre: DEBUG MARKER: MDS: 2.17.51.2 [ 334.173437] Lustre: DEBUG MARKER: OSS: 2.17.51.2 [ 335.742891] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Mar 15 16:36:47 EDT 2026 [ 356.338887] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 358.021568] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 375.264876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.270321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.273252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.293279] Lustre: Skipped 2 previous similar messages [ 380.388653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.408502] Lustre: Skipped 6 previous similar messages [ 381.034648] Lustre: server umount lustre-MDT0000 complete [ 385.809513] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607058 with bad export cookie 10711569280576342475 [ 385.823791] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.848498] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 390.625906] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.626707] LustreError: 9936:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.626894] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.646648] Lustre: Skipped 2 previous similar messages [ 390.683351] LustreError: 9936:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 395.755699] LustreError: 12109:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.760517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.781340] LustreError: 12109:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 395.811519] Lustre: Skipped 2 previous similar messages [ 396.770145] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.809065] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 400.352605] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 400.638763] Lustre: server umount lustre-MDT0001 complete [ 405.277010] Lustre: server umount lustre-OST0000 complete [ 410.290970] Lustre: server umount lustre-OST0001 complete [ 419.037108] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 431.269695] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 471.380352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.877444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 471.968708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.178356] Lustre: lustre-MDT0000: new disk, initializing [ 472.264790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.289309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 477.284474] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 489.669351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.876542] Lustre: 14998:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 489.964318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 489.973506] Lustre: Skipped 1 previous similar message [ 490.135591] Lustre: lustre-MDT0001: new disk, initializing [ 490.318742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 490.383293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 490.399892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 495.094405] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 504.180779] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 510.560848] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 518.368293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.747785] Lustre: lustre-OST0000: new disk, initializing [ 518.750936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 518.889784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.434644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 520.456013] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 520.669616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 525.913796] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 536.565165] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 545.051782] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 545.463797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 551.258975] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 551.603789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 556.517498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.523747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.552909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.558299] Lustre: Skipped 1 previous similar message [ 556.583804] Lustre: Skipped 3 previous similar messages [ 560.117073] Lustre: server umount lustre-OST0000 complete [ 566.754604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.759070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.770244] Lustre: Skipped 1 previous similar message [ 570.377023] Lustre: server umount lustre-MDT0000 complete [ 571.873116] LustreError: 15008:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.891249] LustreError: 15008:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 574.230823] LustreError: 14989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607246 with bad export cookie 10711569280576344008 [ 574.231294] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.253281] LustreError: 14989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 574.753777] Lustre: server umount lustre-MDT0001 complete [ 578.671749] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:40:49 (1773607249) [ 581.203431] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 583.156748] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:40:54 (1773607254) [ 590.147501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.559626] LustreError: 19003:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.617316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.168990] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 601.062610] LustreError: 19004:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.084518] LustreError: 19004:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 605.187463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.800590] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 619.377741] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 623.608519] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 631.618861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.450383] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 645.209044] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 656.866494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 656.871813] LustreError: Skipped 1 previous similar message [ 656.873887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.883178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 656.886233] Lustre: Skipped 1 previous similar message [ 659.950267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 662.221604] Lustre: server umount lustre-OST0000 complete [ 670.184053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.196853] Lustre: Skipped 1 previous similar message [ 673.930453] Lustre: server umount lustre-MDT0000 complete [ 675.298498] LustreError: 19000:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.312140] LustreError: 19000:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 677.936569] LustreError: 18983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607350 with bad export cookie 10711569280576344897 [ 677.938651] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.958899] LustreError: 18983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 678.471590] Lustre: server umount lustre-MDT0001 complete [ 689.045585] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 691.925675] Key type lgssc unregistered [ 692.355857] LNet: 22185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.369280] LNetError: 22185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.420301] LNet: Removed LNI 192.168.203.134@tcp [ 694.481343] Key type .llcrypt unregistered [ 694.484699] Key type ._llcrypt unregistered [ 721.579412] Key type ._llcrypt registered [ 721.585813] Key type .llcrypt registered [ 721.704136] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 723.119978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 723.169938] alg: No test for adler32 (adler32-zlib) [ 724.405309] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 724.844668] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 726.640854] Key type lgssc registered [ 728.381707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 737.002398] loop: module loaded [ 737.134556] loop0: detected capacity change from 0 to 204800000 [ 737.358296] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.386870] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 737.554038] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.257401] loop0: detected capacity change from 0 to 204800000 [ 744.407509] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.414029] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.424944] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.476777] loop0: detected capacity change from 0 to 204800000 [ 759.150534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.208186] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.541968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 760.588449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 760.644735] Lustre: lustre-MDT0000: new disk, initializing [ 760.728143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.766219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 764.268849] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 775.065159] loop1: detected capacity change from 0 to 204800000 [ 780.444566] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 780.763212] Lustre: lustre-OST07c6: new disk, initializing [ 780.767317] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 780.774137] Lustre: Skipped 1 previous similar message [ 780.846594] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 787.768601] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 788.616337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 788.625938] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 788.674229] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 793.482292] Lustre: 24350:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 793.497429] Lustre: 24350:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 793.509225] Lustre: 24350:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 793.522662] Lustre: 24350:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 793.535954] Lustre: 24350:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 793.542535] Lustre: 24350:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 812.623771] Lustre: 25353:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 812.631755] Lustre: 25353:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 812.636851] Lustre: 25353:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.645670] Lustre: 25353:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 812.651241] Lustre: 25353:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 812.657617] Lustre: 25353:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 817.635485] Lustre: 24350:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 817.651173] Lustre: 24350:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 817.657256] Lustre: 24350:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 817.663280] Lustre: 24350:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.670580] Lustre: 24350:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 817.675539] Lustre: 24350:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.683190] Lustre: 24350:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 817.693105] Lustre: 24350:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.700565] Lustre: 24350:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 817.709329] Lustre: 24350:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.717049] Lustre: 24350:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 817.728487] Lustre: 24350:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.959863] Lustre: 25353:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 820.976859] Lustre: 25353:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 820.989340] Lustre: 25353:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.001315] Lustre: 25353:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.019042] Lustre: 25353:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.039637] Lustre: 25353:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.048256] Lustre: 25353:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 821.071077] Lustre: 25353:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.086798] Lustre: 25353:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 821.101712] Lustre: 25353:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.108692] Lustre: 25353:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.115930] Lustre: 25353:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.747597] Lustre: 24349:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 825.762987] Lustre: 24349:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 825.773207] Lustre: 24349:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 825.781230] Lustre: 24349:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.788367] Lustre: 24349:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 825.793834] Lustre: 24349:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.799473] Lustre: 24349:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 825.804693] Lustre: 24349:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.810379] Lustre: 24349:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 825.820208] Lustre: 24349:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.830347] Lustre: 24349:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 825.842340] Lustre: 24349:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.584316] Lustre: 24347:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 837.618415] Lustre: 24347:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 837.639053] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 837.653484] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 837.664958] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 837.670826] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 837.677988] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 837.686560] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 837.695431] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 837.706679] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 837.717400] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 837.728825] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 885.732773] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.739957] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.750764] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.912021] Lustre: server umount lustre-OST07c6 complete [ 896.551507] Lustre: server umount lustre-MDT0000 complete [ 916.050276] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 920.151808] Key type lgssc unregistered [ 920.581592] LNet: 27870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.591602] LNetError: 27870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.606899] LNet: Removed LNI 192.168.203.134@tcp [ 922.001827] Key type .llcrypt unregistered [ 922.004141] Key type ._llcrypt unregistered [ 941.213925] Key type ._llcrypt registered [ 941.215590] Key type .llcrypt registered [ 941.348848] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 942.332658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 942.387665] alg: No test for adler32 (adler32-zlib) [ 943.700277] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 944.121795] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 945.856212] Key type lgssc registered [ 947.290596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.660824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.685068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.338045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.628189] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 968.678391] LustreError: 29124:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.712470] LustreError: 29124:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 973.801609] LustreError: 29123:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.548146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.171506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.848954] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 986.075759] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 989.481681] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 995.789852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.156531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.018299] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1004.400740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1007.794672] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1024.996211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.006766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.021666] Lustre: Skipped 1 previous similar message [ 1025.049606] Lustre: Skipped 1 previous similar message [ 1027.506898] Lustre: server umount lustre-OST0000 complete [ 1031.649466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1031.660280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.682924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.233680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.243638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.587673] Lustre: server umount lustre-MDT0000 complete [ 1040.353592] LustreError: 29124:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.375111] LustreError: 29124:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1043.749864] LustreError: 29103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607716 with bad export cookie 11141439954086957372 [ 1043.757392] LustreError: 29103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1043.757686] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.186505] Lustre: server umount lustre-MDT0001 complete [ 1057.514700] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1062.513314] Key type lgssc unregistered [ 1062.890599] LNet: 32508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.898752] LNetError: 32508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.918166] LNet: Removed LNI 192.168.203.134@tcp [ 1065.291275] Key type .llcrypt unregistered [ 1065.298955] Key type ._llcrypt unregistered [ 1089.793919] Key type ._llcrypt registered [ 1089.801040] Key type .llcrypt registered [ 1089.957581] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1091.433364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1091.469699] alg: No test for adler32 (adler32-zlib) [ 1092.778194] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1093.122354] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1094.873696] Key type lgssc registered [ 1096.274286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1105.179107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1105.216486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.971398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.466893] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1117.156217] LustreError: 33764:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.180246] LustreError: 33764:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1120.209527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.691286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.264197] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1130.825720] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1135.388881] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1143.949726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.376094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.532858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1151.550025] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1158.048659] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1171.951506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.979923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.986040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1177.062523] LustreError: 35708:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.352735] Lustre: server umount lustre-OST0000 complete [ 1182.177087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1182.180722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.186341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1182.186350] Lustre: Skipped 1 previous similar message [ 1182.216793] Lustre: Skipped 2 previous similar messages [ 1187.299832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1187.302660] Lustre: Skipped 1 previous similar message [ 1188.201330] Lustre: server umount lustre-MDT0000 complete [ 1192.419924] LustreError: 33759:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1192.444309] LustreError: 33759:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1192.517724] LustreError: 35381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773607865 with bad export cookie 14766773498532291631 [ 1192.522192] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.530741] LustreError: 35381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1193.022586] Lustre: server umount lustre-MDT0001 complete [ 1206.088278] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1209.069516] Key type lgssc unregistered [ 1209.406747] LNet: 36958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.419244] LNetError: 36958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1209.443591] LNet: Removed LNI 192.168.203.134@tcp [ 1210.387133] Key type .llcrypt unregistered [ 1210.388772] Key type ._llcrypt unregistered [ 1236.688352] Key type ._llcrypt registered [ 1236.693069] Key type .llcrypt registered [ 1236.861819] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1238.999152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1239.082931] alg: No test for adler32 (adler32-zlib) [ 1240.460528] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1240.869992] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1242.608255] Key type lgssc registered [ 1244.246846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1254.301593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1254.323536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.924170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.569702] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1266.154541] LustreError: 38359:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.186365] LustreError: 38359:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1269.474834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.137138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.256768] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1282.388775] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.028329] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1294.983039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1295.339331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.483235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1300.975564] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1306.324828] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1324.374937] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 1325.034689] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.050482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.066614] Lustre: Skipped 1 previous similar message [ 1328.905613] Lustre: server umount lustre-MDT0000 complete [ 1329.480815] LustreError: 40518:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.683359] LustreError: 38338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608005 with bad export cookie 15030125017350343752 [ 1332.693879] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.695750] LustreError: 38338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1334.595149] LustreError: 38355:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.616347] LustreError: 38355:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1334.630675] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 1334.633502] Lustre: Skipped 2 previous similar messages [ 1335.267261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.278693] Lustre: Skipped 1 previous similar message [ 1337.313649] LustreError: 38355:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.314041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1337.341121] LustreError: 38355:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1337.395705] Lustre: Skipped 1 previous similar message [ 1339.197725] Lustre: server umount lustre-MDT0001 complete [ 1355.172318] Lustre: server umount lustre-OST0000 complete [ 1370.521728] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1372.851819] Key type lgssc unregistered [ 1373.147875] LNet: 41892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.156962] LNetError: 41892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.187598] LNet: Removed LNI 192.168.203.134@tcp [ 1374.984752] Key type .llcrypt unregistered [ 1374.990297] Key type ._llcrypt unregistered [ 1395.852759] Key type ._llcrypt registered [ 1395.854792] Key type .llcrypt registered [ 1395.970157] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1396.875754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.992120] alg: No test for adler32 (adler32-zlib) [ 1398.045257] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1398.468038] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1400.136171] Key type lgssc registered [ 1401.285329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1409.436856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1409.451702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.854541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.708704] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1421.295343] LustreError: 43271:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.321406] LustreError: 43271:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1422.701872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.942990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.271695] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1433.376319] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1437.054737] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1444.231024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.790824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.126840] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1456.074668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1458.817341] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1466.343807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.345728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.349482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.373665] Lustre: Skipped 1 previous similar message [ 1471.473174] LustreError: 44864:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.776716] Lustre: server umount lustre-OST0000 complete [ 1489.934366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.967857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.970629] Lustre: Skipped 2 previous similar messages [ 1494.497510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1494.501036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.523101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.414635] Lustre: server umount lustre-MDT0000 complete [ 1497.059904] LustreError: 43271:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.087413] LustreError: 43271:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1500.024139] LustreError: 43253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608172 with bad export cookie 11745278274954244386 [ 1500.032826] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.043546] LustreError: 43253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1506.536250] Lustre: server umount lustre-MDT0001 complete [ 1516.807438] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1519.460728] Key type lgssc unregistered [ 1519.768177] LNet: 46627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.780251] LNetError: 46627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.806750] LNet: Removed LNI 192.168.203.134@tcp [ 1520.568403] Key type .llcrypt unregistered [ 1520.574081] Key type ._llcrypt unregistered [ 1548.219676] Key type ._llcrypt registered [ 1548.222712] Key type .llcrypt registered [ 1548.327919] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1549.983118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.171108] alg: No test for adler32 (adler32-zlib) [ 1551.262524] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1551.636447] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1553.360398] Key type lgssc registered [ 1554.814826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1563.584460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1563.612989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.275746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.591678] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1575.399824] LustreError: 48028:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.433519] LustreError: 48028:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1579.228617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.668460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1583.465493] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1589.896673] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1594.983485] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1602.706539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1603.238355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.888304] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1610.422617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1614.386293] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.443353] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1624.028979] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1636.793695] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-1516h58m26s [ 1638.546189] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1640.929782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1640.939331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.966410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.026848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.033825] Lustre: Skipped 4 previous similar messages [ 1643.901767] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 1643.909979] Lustre: Skipped 1 previous similar message [ 1646.916061] Lustre: server umount lustre-MDT0000 complete [ 1647.941701] LustreError: 50183:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.444675] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1654.095994] LustreError: 48023:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.112760] LustreError: 48023:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 1657.454023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.540155] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.719671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.768800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1658.743551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1661.924598] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1665.145231] Lustre: DEBUG MARKER: Wait for df (36099) ... [ 1666.465648] Lustre: DEBUG MARKER: done [ 1672.166786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1672.177465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.189711] Lustre: Skipped 2 previous similar messages [ 1672.202746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.198091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.207195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.093994] Lustre: server umount lustre-OST0000 complete [ 1684.452121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.452554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.489562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.498139] Lustre: Skipped 4 previous similar messages [ 1686.392832] Lustre: server umount lustre-MDT0000 complete [ 1689.983057] LustreError: 48008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608362 with bad export cookie 569933138752713352 [ 1689.987311] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1689.989735] LustreError: 48008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1690.650964] Lustre: server umount lustre-MDT0001 complete [ 1699.926264] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1702.399622] Key type lgssc unregistered [ 1702.660232] LNet: 52590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.674589] LNetError: 52590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.699299] LNet: Removed LNI 192.168.203.134@tcp [ 1703.576916] Key type .llcrypt unregistered [ 1703.580113] Key type ._llcrypt unregistered [ 1740.351841] Key type ._llcrypt registered [ 1740.359692] Key type .llcrypt registered [ 1740.515975] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1741.742261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.076384] alg: No test for adler32 (adler32-zlib) [ 1743.164736] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1743.402586] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1745.096173] Key type lgssc registered [ 1746.333606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1753.933916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1753.959389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.368666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1755.389948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1755.604191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.784969] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1768.701019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.844064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1768.910911] Lustre: 54762:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1769.319745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1774.277204] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1779.852759] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1783.372389] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1791.711728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1791.859814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1792.005961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.362871] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1800.349859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1803.586546] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1810.394821] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-1517h01m21s [ 1811.971589] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1818.884090] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1821.073852] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1821.140922] LustreError: 54770:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1841.539459] Lustre: lustre-MDT0000: Client a11c2516-ba36-4f35-b8cc-db15045e74d9 (at 192.168.203.34@tcp) reconnecting [ 1861.216558] LustreError: 54770:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1863.538370] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1865.985659] Lustre: DEBUG MARKER: done [ 1872.612733] Lustre: server umount lustre-OST0000 complete [ 1877.474139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.501447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1878.584521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1878.587412] Lustre: Skipped 1 previous similar message [ 1884.641800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.651959] Lustre: Skipped 1 previous similar message [ 1888.986286] Lustre: server umount lustre-MDT0000 complete [ 1892.832463] Lustre: 54085:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608549/real 1773608549] req@00000000b1652956 x1859763217420800/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773608565 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1892.883392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.894936] Lustre: Skipped 1 previous similar message [ 1893.605469] LustreError: 54755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608566 with bad export cookie 12532145620584445255 [ 1893.607337] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.612952] LustreError: 54755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1894.054878] Lustre: server umount lustre-MDT0001 complete [ 1905.897894] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1909.697446] Key type lgssc unregistered [ 1910.070091] LNet: 58749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.074556] LNetError: 58749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.142380] LNet: Removed LNI 192.168.203.134@tcp [ 1912.005503] Key type .llcrypt unregistered [ 1912.007262] Key type ._llcrypt unregistered [ 1959.887698] Key type ._llcrypt registered [ 1959.890126] Key type .llcrypt registered [ 1959.989763] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1961.255034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.282176] alg: No test for adler32 (adler32-zlib) [ 1962.483852] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 1962.773450] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1964.450358] Key type lgssc registered [ 1965.458745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1968.386515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1968.429853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.830875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1969.887779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1970.143229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.053482] Lustre: Failing over lustre-MDT0000 [ 1971.415342] Lustre: server umount lustre-MDT0000 complete [ 1977.695972] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:04:09 (1773608649) [ 1984.092879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.453982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.169668] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1998.315726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.533536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1998.594804] Lustre: 61536:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1998.994058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2004.341649] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2010.550819] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2015.593520] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2022.008626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.160992] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2022.346208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.621822] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2028.158283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2033.135395] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2039.457609] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2048.996371] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.003398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.016358] Lustre: Skipped 1 previous similar message [ 2049.027795] Lustre: Skipped 2 previous similar messages [ 2052.367442] Lustre: server umount lustre-MDT0000 complete [ 2054.124342] LustreError: 61546:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.144321] LustreError: 61546:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2055.833951] LustreError: 61526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608728 with bad export cookie 6021785429230164596 [ 2055.843405] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.847430] LustreError: 61526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2056.083522] Lustre: server umount lustre-MDT0001 complete [ 2059.271734] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2064.353284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2073.622450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.592692] Lustre: 60456:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608731/real 1773608731] req@0000000086c1e965 x1859763448093824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773608747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.622793] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.658198] Lustre: Skipped 1 previous similar message [ 2075.616262] Lustre: 60455:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608732/real 1773608732] req@00000000fcd66ced x1859763448093952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773608748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2079.712204] Lustre: 60458:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608736/real 1773608736] req@00000000dde86490 x1859763448094080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773608752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2080.801229] LustreError: 60454:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dde86490 x1859763448094848/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.104226] LustreError: 63156:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.223733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.436564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2082.144589] Lustre: 60457:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608738/real 1773608738] req@000000000a644b7d x1859763448094336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773608754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2083.241412] LustreError: 63157:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.301769] LustreError: 64965:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.328067] LustreError: 64965:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2085.453625] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2088.417742] Lustre: 60455:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608744/real 1773608744] req@00000000d68f39ee x1859763448094464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773608760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.474338] LustreError: 64967:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.498984] LustreError: 64967:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2094.503953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.783251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2098.902730] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2100.216281] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2104.360916] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2106.226802] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.108733] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2119.651423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.658491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.689603] Lustre: Skipped 1 previous similar message [ 2119.715038] Lustre: Skipped 2 previous similar messages [ 2124.773975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.105243] Lustre: server umount lustre-MDT0000 complete [ 2129.901592] LustreError: 64965:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.917636] LustreError: 64965:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2130.155526] LustreError: 61526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608802 with bad export cookie 6021785429230166878 [ 2130.156703] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.170068] LustreError: 61526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2130.704718] Lustre: server umount lustre-MDT0001 complete [ 2137.036916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2146.219238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.395248] Lustre: 60458:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773608807/real 1773608807] req@0000000056152811 x1859763448120960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773608823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2151.438597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.453918] Lustre: Skipped 1 previous similar message [ 2154.528926] LustreError: 60454:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005209e166 x1859763448121728/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.787227] LustreError: 65684:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.882874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.933692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2159.638834] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2159.977471] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2168.474713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.515461] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2174.463133] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2178.871726] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2182.643288] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2194.405570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.410194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.426147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.430525] Lustre: Skipped 2 previous similar messages [ 2196.404545] Lustre: server umount lustre-MDT0000 complete [ 2200.053491] LustreError: 63156:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.078530] LustreError: 63156:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 2200.140283] LustreError: 61528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773608872 with bad export cookie 6021785429230168257 [ 2200.149954] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.150231] LustreError: 61528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2200.603708] Lustre: server umount lustre-MDT0001 complete [ 2207.073055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2210.326610] Lustre: DEBUG MARKER: files compared the same [ 2222.482412] Lustre: server umount lustre-OST0000 complete [ 2239.606313] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2242.519446] Key type lgssc unregistered [ 2242.952542] LNet: 70466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.956397] LNetError: 70466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.997374] LNet: Removed LNI 192.168.203.134@tcp [ 2243.774492] Key type .llcrypt unregistered [ 2243.777349] Key type ._llcrypt unregistered [ 2268.527437] Key type ._llcrypt registered [ 2268.533309] Key type .llcrypt registered [ 2268.629872] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2270.136194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.185045] alg: No test for adler32 (adler32-zlib) [ 2271.520489] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2271.860735] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2273.584200] Key type lgssc registered [ 2274.803593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.163855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.189418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.556687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.887760] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 2294.762121] LustreError: 71870:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.790439] LustreError: 71870:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2297.370782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.864906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2302.590711] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 2307.572963] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2311.278968] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2317.501564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.738182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.733281] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug +malloc all [ 2329.016974] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.100040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2338.279210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2338.286308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.310146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.321340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.323678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.357439] Lustre: Skipped 1 previous similar message [ 2345.455181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2345.466566] Lustre: Skipped 1 previous similar message [ 2350.566020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.581927] Lustre: Skipped 1 previous similar message [ 2352.096182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2352.352353] Lustre: server umount lustre-OST0000 complete [ 2359.266152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2359.278886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.300359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.336237] Lustre: server umount lustre-MDT0000 complete [ 2365.923145] LustreError: 71866:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.940453] LustreError: 71866:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2366.117019] LustreError: 71850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609038 with bad export cookie 338186349498643 [ 2366.125692] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.650726] Lustre: server umount lustre-MDT0001 complete [ 2377.543608] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2380.382699] Key type lgssc unregistered [ 2380.594061] LNet: 75057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.599946] LNetError: 75057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.671675] LNet: Removed LNI 192.168.203.134@tcp [ 2382.669088] Key type .llcrypt unregistered [ 2382.671330] Key type ._llcrypt unregistered [ 2402.974495] Key type ._llcrypt registered [ 2402.976648] Key type .llcrypt registered [ 2403.104565] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2404.262392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2404.322649] alg: No test for adler32 (adler32-zlib) [ 2405.299281] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2405.517334] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2407.249213] Key type lgssc registered [ 2408.220693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.662753] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.680974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.304346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2435.425878] LustreError: 76411:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 2462.177568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.120255] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2473.721192] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2480.581942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.682139] Lustre: *** cfs_fail_loc=706, val=0*** [ 2481.022424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.110945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2484.338706] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2486.242648] LustreError: 76437:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.277318] LustreError: 76437:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2491.363349] LustreError: 76436:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.750627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.166207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2498.013788] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2503.196219] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2507.149967] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2509.797186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2509.814051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.829691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.841280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.850342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.863311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.873590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.879111] Lustre: Skipped 1 previous similar message [ 2518.008198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2518.020530] Lustre: Skipped 1 previous similar message [ 2524.132316] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2524.276630] Lustre: server umount lustre-OST0000 complete [ 2530.283516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2530.290327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.308694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2530.319621] Lustre: Skipped 2 previous similar messages [ 2534.469715] Lustre: server umount lustre-MDT0000 complete [ 2536.421394] LustreError: 77386:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.434610] LustreError: 77386:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2537.557072] LustreError: 76419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609210 with bad export cookie 16385434185302954542 [ 2537.568049] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.984050] Lustre: server umount lustre-MDT0001 complete [ 2546.064357] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2548.292850] Key type lgssc unregistered [ 2548.540863] LNet: 79671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.550503] LNetError: 79671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.570540] LNet: Removed LNI 192.168.203.134@tcp [ 2549.282482] Key type .llcrypt unregistered [ 2549.285631] Key type ._llcrypt unregistered [ 2574.503366] Key type ._llcrypt registered [ 2574.511397] Key type .llcrypt registered [ 2574.741858] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2576.292915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2576.368034] alg: No test for adler32 (adler32-zlib) [ 2577.442939] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2577.644052] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2579.338344] Key type lgssc registered [ 2581.008032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2589.637856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2589.671571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.111875] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2602.372049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.176810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2607.768782] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2608.617439] LustreError: 81746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.648377] LustreError: 81746:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2612.709302] LustreError: 81750:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.729188] LustreError: 81750:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2615.815047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.134423] LustreError: 81746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.183559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.409802] LustreError: 81749:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.431959] LustreError: 81749:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2622.094701] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2626.535203] LustreError: 81750:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.573286] LustreError: 81750:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2630.908708] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.937261] Lustre: MGS: Client 1d374946-27d7-4b71-ba00-81a4aa6d69e1 (at 0@lo) reconnecting [ 2630.971544] LustreError: 81760:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2631.293960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.376623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2635.861940] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2651.617867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2651.626669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.651039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2652.136437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.182610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2652.199137] Lustre: Skipped 1 previous similar message [ 2656.604921] Lustre: server umount lustre-OST0000 complete [ 2662.371680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2662.377070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.377626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.389233] Lustre: Skipped 1 previous similar message [ 2662.416020] Lustre: Skipped 1 previous similar message [ 2666.502235] Lustre: server umount lustre-MDT0000 complete [ 2667.489522] LustreError: 81749:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.505822] LustreError: 81749:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2670.326823] Lustre: server umount lustre-MDT0001 complete [ 2673.720777] Lustre: server umount MGS complete [ 2685.783759] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:15:56 (1773609356) [ 2715.712620] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2727.346656] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2765.859135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.530668] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2781.031269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.421774] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2781.492378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2781.631211] Lustre: lustre-MDT0001: new disk, initializing [ 2781.742039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.801574] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2781.812217] Lustre: 88836:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2781.828228] LustreError: 88836:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2785.884139] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2797.049255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2797.364929] Lustre: lustre-OST0000: new disk, initializing [ 2797.370215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2797.499896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.747379] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2808.433336] Lustre: 90024:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a14130c1 x1859764092968320/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773609496 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2812.809695] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2818.703378] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.714395] Lustre: MGS: Client fd0581d8-0f60-4245-b25a-4a472a7b069f (at 0@lo) reconnecting [ 2818.746731] Lustre: 88122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2818.770895] LustreError: 88122:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2818.781612] LustreError: 88122:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2818.798816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2818.857771] Lustre: lustre-MDT0000: new disk, initializing [ 2818.985058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.038594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2819.157896] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2820.129828] Lustre: 90544:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000000e200fe8 x1859764092974848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773609508 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2823.277556] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2827.754610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2827.764337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2829.232165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2829.247687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2829.320712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2853.351231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2853.362915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.383182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.957791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.965697] Lustre: Skipped 2 previous similar messages [ 2859.675785] Lustre: server umount lustre-OST0000 complete [ 2868.193474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.193902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.203649] Lustre: Skipped 1 previous similar message [ 2868.204920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.242642] LustreError: Skipped 1 previous similar message [ 2870.150513] Lustre: server umount lustre-MDT0000 complete [ 2873.313505] LustreError: 88812:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.331543] LustreError: 88812:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2888.160185] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2888.446918] Lustre: server umount lustre-MDT0001 complete [ 2892.082088] Lustre: server umount MGS complete [ 2903.323354] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:19:34 (1773609574) [ 2921.078471] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2924.300178] Key type lgssc unregistered [ 2924.559801] LNet: 93011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2924.569762] LNetError: 93011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2924.589651] LNet: Removed LNI 192.168.203.134@tcp [ 2925.487920] Key type .llcrypt unregistered [ 2925.491751] Key type ._llcrypt unregistered [ 2943.831224] Key type ._llcrypt registered [ 2943.834183] Key type .llcrypt registered [ 2943.944402] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2945.570965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2945.641682] alg: No test for adler32 (adler32-zlib) [ 2946.738719] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 2946.973918] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2948.673359] Key type lgssc registered [ 2950.183179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2959.933954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2959.974847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.556524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.976467] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2972.137694] LustreError: 94219:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.166331] LustreError: 94219:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2975.202619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.703101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.919534] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2989.524327] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2993.941648] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3001.328344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3002.066748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.160323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3008.711867] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3014.217385] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3027.949397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.958219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.980610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.009283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.012096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.731941] Lustre: server umount lustre-OST0000 complete [ 3037.154524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3037.166031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.189458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3037.195828] Lustre: Skipped 1 previous similar message [ 3041.250755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.252259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.174583] Lustre: server umount lustre-MDT0000 complete [ 3046.370511] LustreError: 94214:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.404733] LustreError: 94214:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3047.746534] LustreError: 94201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609720 with bad export cookie 18160333383864023295 [ 3047.771843] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.344805] Lustre: server umount lustre-MDT0001 complete [ 3059.732674] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3062.323550] Key type lgssc unregistered [ 3062.634319] LNet: 97398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3062.645675] LNetError: 97398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3062.680595] LNet: Removed LNI 192.168.203.134@tcp [ 3063.767870] Key type .llcrypt unregistered [ 3063.770103] Key type ._llcrypt unregistered [ 3081.344636] Key type ._llcrypt registered [ 3081.346378] Key type .llcrypt registered [ 3081.457644] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3082.840401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3083.074074] alg: No test for adler32 (adler32-zlib) [ 3084.242796] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3084.528332] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3086.218708] Key type lgssc registered [ 3087.816240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.758476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3094.763407] LustreError: 98468:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3094.766541] Lustre: Skipped 1 previous similar message [ 3094.774032] LustreError: 98469:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3094.778166] LustreError: 98468:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3094.784110] LustreError: 98468:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3094.790861] LustreError: 98468:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3094.795676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.405257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.569846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.832521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3110.013240] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3115.495475] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing lsmod [ 3118.046597] LustreError: 99637:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3118.064660] LustreError: 99640:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3118.070957] LustreError: 99637:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3118.076804] LustreError: 99637:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3118.086574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.087307] LustreError: 99637:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3118.412809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.696899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.706074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.722338] Lustre: Skipped 2 previous similar messages [ 3123.741775] Lustre: Skipped 2 previous similar messages [ 3127.308676] Lustre: server umount lustre-MDT0000 complete [ 3128.812112] LustreError: 98506:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.827114] LustreError: 98506:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 3129.830900] LustreError: 98506:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.847824] LustreError: 98506:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3131.035320] LustreError: 98487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609803 with bad export cookie 4675965281589965320 [ 3131.053248] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.319791] Lustre: server umount lustre-MDT0001 complete [ 3145.502938] Lustre: server umount lustre-OST0000 complete [ 3154.685497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.118742] LustreError: 100706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.205100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.598368] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3160.545595] LustreError: 100707:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.641849] LustreError: 100706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3168.508821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.965081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3173.888233] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3180.047133] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3184.316874] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3191.350383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.606690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3197.666949] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3199.812261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3202.988651] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3210.213534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.214949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.241033] Lustre: Skipped 1 previous similar message [ 3210.267019] Lustre: Skipped 1 previous similar message [ 3215.331753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.348787] Lustre: Skipped 1 previous similar message [ 3215.882246] Lustre: server umount lustre-OST0000 complete [ 3220.452252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.452432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3220.456503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.473745] Lustre: Skipped 1 previous similar message [ 3225.570049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.577831] Lustre: Skipped 2 previous similar messages [ 3234.272218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3234.635390] Lustre: server umount lustre-MDT0000 complete [ 3235.813180] LustreError: 100706:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.830940] LustreError: 100706:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3238.465392] LustreError: 100687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773609911 with bad export cookie 4675965281589966006 [ 3238.477494] LustreError: 100687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3238.497519] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.740288] Lustre: server umount lustre-MDT0001 complete [ 3251.064242] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3254.204224] Key type lgssc unregistered [ 3254.581986] LNet: 103889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.597025] LNetError: 103889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.608664] LNet: Removed LNI 192.168.203.134@tcp [ 3255.447817] Key type .llcrypt unregistered [ 3255.449949] Key type ._llcrypt unregistered [ 3278.351517] Key type ._llcrypt registered [ 3278.354297] Key type .llcrypt registered [ 3278.460028] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3279.931068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3280.100915] alg: No test for adler32 (adler32-zlib) [ 3281.118595] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3281.319691] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3283.009309] Key type lgssc registered [ 3284.398630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3294.018197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3294.074033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.647743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.617423] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3305.977572] LustreError: 105291:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3306.025838] LustreError: 105291:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3311.075143] LustreError: 105292:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.452616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.003703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3319.169776] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3325.566215] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3330.096160] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3336.688672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.178082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.308531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3344.858826] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3351.812669] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3357.434909] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3361.898448] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3366.881626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.888833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.906895] Lustre: Skipped 1 previous similar message [ 3372.013633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.021381] Lustre: Skipped 2 previous similar messages [ 3377.125661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.134604] Lustre: Skipped 1 previous similar message [ 3381.216173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3381.382903] Lustre: server umount lustre-OST0000 complete [ 3386.849666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3386.853786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.869085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.550415] Lustre: server umount lustre-MDT0000 complete [ 3392.483693] LustreError: 105288:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.503672] LustreError: 105288:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3395.106744] LustreError: 105271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610067 with bad export cookie 12414015528300411614 [ 3395.118468] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.119148] LustreError: 105271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3395.475543] Lustre: server umount lustre-MDT0001 complete [ 3405.942236] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3409.272894] Key type lgssc unregistered [ 3409.693653] LNet: 108580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.703085] LNetError: 108580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3409.722929] LNet: Removed LNI 192.168.203.134@tcp [ 3410.771052] Key type .llcrypt unregistered [ 3410.772671] Key type ._llcrypt unregistered [ 3429.262095] Key type ._llcrypt registered [ 3429.265318] Key type .llcrypt registered [ 3429.391679] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3430.986819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3431.044654] alg: No test for adler32 (adler32-zlib) [ 3432.273435] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3432.661685] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3434.385325] Key type lgssc registered [ 3435.707647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3444.126496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3444.138210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.530647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.569735] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3455.974948] LustreError: 109786:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.998079] LustreError: 109786:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3459.698847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.233938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3464.945719] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3471.649883] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3475.918816] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3483.256100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.766446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.926111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3491.367258] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3497.288591] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3506.660601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.661542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3506.665457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.677047] Lustre: Skipped 1 previous similar message [ 3511.782470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.786826] Lustre: Skipped 1 previous similar message [ 3512.313363] Lustre: server umount lustre-OST0000 complete [ 3516.385413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3516.399984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.416459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3516.422885] Lustre: Skipped 1 previous similar message [ 3522.023562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.029368] Lustre: Skipped 2 previous similar messages [ 3522.821805] Lustre: server umount lustre-MDT0000 complete [ 3527.137646] LustreError: 109786:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.157975] LustreError: 109786:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3527.285521] LustreError: 110504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610199 with bad export cookie 98180283172194539 [ 3527.288808] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.297323] LustreError: 110504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3527.852946] Lustre: server umount lustre-MDT0001 complete [ 3540.066220] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3543.048189] Key type lgssc unregistered [ 3543.436600] LNet: 112967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3543.444861] LNetError: 112967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3544.494575] LNet: Removed LNI 192.168.203.134@tcp [ 3545.694820] Key type .llcrypt unregistered [ 3545.696476] Key type ._llcrypt unregistered [ 3574.511309] Key type ._llcrypt registered [ 3574.514168] Key type .llcrypt registered [ 3574.636021] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3576.127891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3576.258237] alg: No test for adler32 (adler32-zlib) [ 3577.454376] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3577.660726] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3579.321041] Key type lgssc registered [ 3580.880722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3589.442827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3589.482962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.923332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.522855] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3601.394179] LustreError: 114368:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.426741] LustreError: 114368:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3605.426861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.849935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3610.553552] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3616.481477] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3621.209903] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3629.207421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3629.576801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.724480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3637.371925] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3643.439582] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3650.137206] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3654.790318] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3654.798357] Lustre: Skipped 2 previous similar messages [ 3657.305654] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3657.356859] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3658.794511] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3658.801793] Lustre: Skipped 2 previous similar messages [ 3667.034107] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3667.049870] Lustre: Skipped 1 previous similar message [ 3671.913288] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3671.923191] Lustre: Skipped 2 previous similar messages [ 3673.455667] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.34@tcp 192.168.0.[2,10]@tcp [ 3685.823651] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.34/24@tcp in log lustre-MDT0000 [ 3685.834296] Lustre: Skipped 2 previous similar messages [ 3693.705154] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.34/24@tcp [ 3693.711679] Lustre: Skipped 1 previous similar message [ 3702.029322] Lustre: server umount lustre-OST0000 complete [ 3707.362378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.380531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.387853] Lustre: Skipped 1 previous similar message [ 3712.483336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3712.496119] Lustre: Skipped 1 previous similar message [ 3717.617077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.633251] Lustre: Skipped 1 previous similar message [ 3718.624577] Lustre: 113680:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610374/real 1773610374] req@00000000318617e0 x1859765141570176/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773610390 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3718.659259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.676294] Lustre: Skipped 1 previous similar message [ 3720.160448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3720.589639] Lustre: server umount lustre-MDT0000 complete [ 3722.784267] Lustre: 113678:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773610379/real 1773610379] req@0000000047dc5a8e x1859765141572480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773610395 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3723.748260] LustreError: 114362:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.789203] LustreError: 114362:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3725.139209] LustreError: 114347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610397 with bad export cookie 15255113303163591997 [ 3725.142606] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3725.149240] LustreError: 114347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3725.525224] Lustre: server umount lustre-MDT0001 complete [ 3738.960156] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3741.983480] Key type lgssc unregistered [ 3742.287375] LNet: 119011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3742.296384] LNetError: 119011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3742.311766] LNet: Removed LNI 192.168.203.134@tcp [ 3743.373852] Key type .llcrypt unregistered [ 3743.377147] Key type ._llcrypt unregistered [ 3772.880580] Key type ._llcrypt registered [ 3772.884536] Key type .llcrypt registered [ 3773.081280] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3774.799784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3774.964440] alg: No test for adler32 (adler32-zlib) [ 3776.132538] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3776.395020] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3778.064255] Key type lgssc registered [ 3779.319383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3787.316997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3787.337226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.920638] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3788.928547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.947465] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.34/24@tcp [ 3793.373577] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3798.501059] LustreError: 120557:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.523236] LustreError: 120557:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3802.095281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.453160] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3802.484454] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.34/24@tcp [ 3802.516305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3806.324258] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3811.627641] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3815.136317] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3822.384172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3822.705428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3828.456759] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3833.845565] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3833.989431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3844.586323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.593993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.598264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.611367] Lustre: Skipped 1 previous similar message [ 3844.633887] Lustre: Skipped 1 previous similar message [ 3849.045202] Lustre: server umount lustre-OST0000 complete [ 3853.796408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3853.804234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.834937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.818042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.835767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.840834] Lustre: Skipped 1 previous similar message [ 3859.937749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.943640] Lustre: Skipped 1 previous similar message [ 3865.061309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.072583] Lustre: Skipped 1 previous similar message [ 3867.104132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3867.312770] Lustre: server umount lustre-MDT0000 complete [ 3870.181966] LustreError: 122468:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.215782] LustreError: 122468:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3871.015628] LustreError: 120537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610543 with bad export cookie 482508240814938310 [ 3871.018556] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.027744] LustreError: 120537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3871.412120] Lustre: server umount lustre-MDT0001 complete [ 3883.650475] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3887.015981] Key type lgssc unregistered [ 3887.337116] LNet: 123793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3887.350955] LNetError: 123793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3887.379976] LNet: Removed LNI 192.168.203.134@tcp [ 3888.364893] Key type .llcrypt unregistered [ 3888.369619] Key type ._llcrypt unregistered [ 3915.413767] Key type ._llcrypt registered [ 3915.416306] Key type .llcrypt registered [ 3915.518256] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 3930.136726] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3931.154342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3931.253026] alg: No test for adler32 (adler32-zlib) [ 3932.352792] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 3932.590951] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3934.336197] Key type lgssc registered [ 3935.564738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3968.235970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3968.291621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.815499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3969.878376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3970.041675] Lustre: lustre-MDT0000: new disk, initializing [ 3970.181658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.200835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3975.377473] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3987.956815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.218653] Lustre: 127169:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3988.348404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3988.354410] Lustre: Skipped 1 previous similar message [ 3988.474958] Lustre: lustre-MDT0001: new disk, initializing [ 3988.674722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3988.737922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3988.748431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3994.260736] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4001.948874] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4005.776139] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4012.438863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.776306] Lustre: lustre-OST0000: new disk, initializing [ 4012.791600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4012.931330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.840462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4019.857577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4020.041062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4020.658665] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4030.083460] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4035.383752] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4035.704407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4040.268887] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4040.458174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4045.799138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.808216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4045.808956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.819409] Lustre: Skipped 1 previous similar message [ 4048.523305] Lustre: server umount lustre-OST0000 complete [ 4055.009406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.018268] LustreError: Skipped 1 previous similar message [ 4055.022441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.029979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4055.040422] Lustre: Skipped 1 previous similar message [ 4056.034192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.038309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4056.059599] Lustre: Skipped 1 previous similar message [ 4058.786841] Lustre: server umount lustre-MDT0000 complete [ 4062.315542] LustreError: 127161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610734 with bad export cookie 17721818529146073826 [ 4062.319367] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.326204] LustreError: 127161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4063.022490] Lustre: server umount lustre-MDT0001 complete [ 4072.265567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.003325] LustreError: 130980:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.038938] LustreError: 130980:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4073.163956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.071579] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4078.564056] LustreError: 130981:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.670705] LustreError: 130980:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.763909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4087.022210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.275310] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4096.939477] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4100.631889] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.286277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.634365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.121760] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4120.379302] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4125.516994] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4125.829982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4131.837261] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4132.113834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4141.941462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.063357] Lustre: lustre-OST0001: new disk, initializing [ 4142.068436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4142.160290] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.678745] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4152.444229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4152.457310] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4152.630706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4155.916175] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4156.222843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4160.909622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4161.115385] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4165.140187] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90fd06335800.ost_server_uuid 50 [ 4166.740562] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90fd06335800.ost_server_uuid in FULL state after 0 sec [ 4176.331925] Lustre: server umount lustre-OST0001 complete [ 4178.409232] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4178.433392] Lustre: Skipped 1 previous similar message [ 4178.461230] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4178.463732] LustreError: 135139:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.507822] LustreError: 135139:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4183.523846] LustreError: 132574:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.533592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4183.541406] LustreError: 132574:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4183.543535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.548290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4183.608493] Lustre: Skipped 1 previous similar message [ 4188.647210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4193.769317] LustreError: 132575:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.799472] LustreError: 132575:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4194.272286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4194.380739] Lustre: server umount lustre-OST0000 complete [ 4198.902361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.922224] Lustre: Skipped 1 previous similar message [ 4198.930245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4198.934347] Lustre: Skipped 4 previous similar messages [ 4204.729275] Lustre: server umount lustre-MDT0000 complete [ 4208.137849] LustreError: 130960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773610880 with bad export cookie 17721818529146074715 [ 4208.146907] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4208.154385] LustreError: 130960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4208.696497] Lustre: server umount lustre-MDT0001 complete [ 4227.358924] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4230.484552] Key type lgssc unregistered [ 4230.771971] LNet: 136822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4230.779121] LNetError: 136822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4231.850254] LNet: Removed LNI 192.168.203.134@tcp [ 4232.834603] Key type .llcrypt unregistered [ 4232.837495] Key type ._llcrypt unregistered [ 4273.760390] Key type ._llcrypt registered [ 4273.763049] Key type .llcrypt registered [ 4273.869725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4287.549891] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4288.497085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4288.530805] alg: No test for adler32 (adler32-zlib) [ 4289.651963] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 4289.888955] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4291.592382] Key type lgssc registered [ 4292.891867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4326.236354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4326.264536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.659937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4327.730894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4327.820845] Lustre: lustre-MDT0000: new disk, initializing [ 4327.905064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.958639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4332.745579] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4343.927758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.189756] Lustre: 140943:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4344.298927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4344.306035] Lustre: Skipped 1 previous similar message [ 4344.460208] Lustre: lustre-MDT0001: new disk, initializing [ 4344.567618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4344.637462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4344.652636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4349.841395] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4359.151132] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4363.664331] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4371.715151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4372.177434] Lustre: lustre-OST0000: new disk, initializing [ 4372.190392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4372.317492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.660930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4381.341698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4381.363914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4381.509890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4388.647098] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4393.991788] Lustre: Failing over lustre-OST0000 [ 4396.019075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4396.029896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.047244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4396.681531] Lustre: server umount lustre-OST0000 complete [ 4413.417721] Lustre: 139168:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611069/real 1773611069] req@000000005bfd66c3 x1859765888123776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773611085 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4413.488258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.357911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4417.791050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.813677] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4418.594775] Lustre: 139169:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611074/real 1773611074] req@00000000b6aa5707 x1859765888125568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773611090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4423.139742] Lustre: 139169:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611079/real 1773611079] req@00000000dde86490 x1859765888126592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773611095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4423.922137] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4424.356624] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4428.002956] Lustre: 139169:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611084/real 1773611084] req@0000000047dc5a8e x1859765888127616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773611100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4428.837077] Lustre: Failing over lustre-MDT0000 [ 4429.130451] Lustre: server umount lustre-MDT0000 complete [ 4429.281631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.315369] Lustre: Skipped 1 previous similar message [ 4429.332733] LustreError: 141967:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.370130] LustreError: 141967:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 4432.226193] LustreError: 140951:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.403875] LustreError: 140957:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.440898] LustreError: 140957:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4439.522852] LustreError: 141967:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.549908] LustreError: 141967:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4444.641782] Lustre: 139171:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773611101/real 1773611101] req@00000000b6aa5707 x1859765888136448/t0(0) o400->MGC192.168.203.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773611117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4444.648667] LustreError: 140957:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.665542] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4444.694420] LustreError: 140957:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4448.990479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.689558] LustreError: 140952:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.700650] LustreError: 140952:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 4454.885572] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x73854e6252da24a2 [ 4454.896544] Lustre: MGC192.168.203.134@tcp: Connection restored to 0@lo (at 0@lo) [ 4455.155030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4455.223058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.247266] Lustre: lustre-OST0000: Recovery over after 0:32, of 3 clients 3 recovered and 0 were evicted. [ 4455.325058] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4457.279673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4460.538659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4460.559429] Lustre: Skipped 2 previous similar messages [ 4460.596740] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4460.664918] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4460.723402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4475.893429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4475.894661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.906450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4475.922243] Lustre: Skipped 2 previous similar messages [ 4475.934907] Lustre: Skipped 1 previous similar message [ 4477.621935] Lustre: server umount lustre-OST0000 complete [ 4486.113883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.128665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.237363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.252265] Lustre: Skipped 2 previous similar messages [ 4496.353043] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4496.355200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4496.387342] Lustre: Skipped 1 previous similar message [ 4496.845747] Lustre: server umount lustre-MDT0000 complete [ 4501.370463] LustreError: 140936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611173 with bad export cookie 8324145670468019362 [ 4501.397811] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.789488] Lustre: server umount lustre-MDT0001 complete [ 4513.385713] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4516.404985] Key type lgssc unregistered [ 4516.710640] LNet: 146191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4516.720966] LNetError: 146191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4516.737272] LNet: Removed LNI 192.168.203.134@tcp [ 4517.575454] Key type .llcrypt unregistered [ 4517.578680] Key type ._llcrypt unregistered [ 4555.138837] Key type ._llcrypt registered [ 4555.141648] Key type .llcrypt registered [ 4555.240401] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4556.510820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4556.801091] alg: No test for adler32 (adler32-zlib) [ 4558.215756] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 4558.613517] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4560.392402] Key type lgssc registered [ 4561.807354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4570.179954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4570.230116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.922079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.023688] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4582.371659] LustreError: 147695:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.399771] LustreError: 147695:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4584.773197] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4587.501056] LustreError: 147696:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4589.431705] LustreError: 147724:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4589.443617] LustreError: 147724:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4589.461662] LustreError: 147724:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4590.132790] Lustre: server umount lustre-MDT0000 complete [ 4605.980500] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4615.601242] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4649.135370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.386416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4649.418507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4649.486830] Lustre: lustre-MDT0000: new disk, initializing [ 4649.563816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.583414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4653.892672] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4664.195310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.368393] Lustre: 150880:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4664.428403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4664.433577] Lustre: Skipped 1 previous similar message [ 4664.504335] Lustre: lustre-MDT0001: new disk, initializing [ 4664.586855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4664.628959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4664.648228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4669.002233] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4676.583851] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4680.787124] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4688.009036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.372754] Lustre: lustre-OST0000: new disk, initializing [ 4688.379221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4688.479661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.058834] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4696.171567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4696.204322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4696.411490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4703.873902] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4709.084962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4711.169820] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:49:42 (1773611382) [ 4925.921442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4925.934269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.967349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.944835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.945696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.974259] Lustre: Skipped 2 previous similar messages [ 4932.065541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4937.185448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4937.198584] Lustre: Skipped 4 previous similar messages [ 4939.232640] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4939.449474] Lustre: server umount lustre-MDT0000 complete [ 4942.323679] LustreError: 151741:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.026573] LustreError: 150873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611615 with bad export cookie 12147662316133983843 [ 4943.031396] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4943.033233] LustreError: 150873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4947.425064] LustreError: 150887:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.426843] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.451638] LustreError: 150887:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4947.465831] Lustre: Skipped 1 previous similar message [ 4947.469061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4949.554325] Lustre: server umount lustre-MDT0001 complete [ 4959.741951] Lustre: server umount lustre-OST0000 complete [ 4970.429376] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4980.872600] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5020.993534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.224209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5021.301903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5021.478755] Lustre: lustre-MDT0000: new disk, initializing [ 5021.613455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.633874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5026.571312] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5036.934787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.076511] Lustre: 157135:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5037.153611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5037.158838] Lustre: Skipped 1 previous similar message [ 5037.254602] Lustre: lustre-MDT0001: new disk, initializing [ 5037.357728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5037.414484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5037.437540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5042.500830] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5050.326768] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5055.688257] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5064.238310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.590783] Lustre: lustre-OST0000: new disk, initializing [ 5064.605585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5064.773552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.058938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5073.071687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5073.086818] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5073.175022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5081.711110] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5086.577892] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5086.833830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5092.429833] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5092.646556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5098.980513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5098.984939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.004659] LustreError: Skipped 1 previous similar message [ 5099.035892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5099.046082] Lustre: Skipped 2 previous similar messages [ 5100.820897] Lustre: server umount lustre-OST0000 complete [ 5109.219943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5109.231272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.254542] Lustre: Skipped 2 previous similar messages [ 5110.994657] Lustre: server umount lustre-MDT0000 complete [ 5114.345192] LustreError: 159091:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5114.368679] LustreError: 159091:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5114.492760] LustreError: 157126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611787 with bad export cookie 12147662316134142673 [ 5114.500599] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.509948] LustreError: 157126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5114.998942] Lustre: server umount lustre-MDT0001 complete [ 5123.296589] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:56:34 (1773611794) [ 5138.423280] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5147.158454] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5181.377165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.729410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5181.767848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5181.911854] Lustre: lustre-MDT0000: new disk, initializing [ 5182.043054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.082393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5186.583866] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5197.073857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.175663] Lustre: 163454:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5197.309349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5201.730112] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5209.346415] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5212.851199] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5219.118560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5219.339801] Lustre: lustre-OST0000: new disk, initializing [ 5219.342647] Lustre: Skipped 1 previous similar message [ 5219.346423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5219.352486] Lustre: Skipped 2 previous similar messages [ 5225.065877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5225.078472] Lustre: Skipped 1 previous similar message [ 5225.080675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5225.210299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5225.405605] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5233.090445] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5240.815445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5240.816255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5240.823041] LustreError: Skipped 1 previous similar message [ 5240.824329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5240.824335] Lustre: Skipped 2 previous similar messages [ 5240.847339] Lustre: Skipped 1 previous similar message [ 5246.485244] Lustre: server umount lustre-OST0000 complete [ 5265.376120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5265.837847] Lustre: server umount lustre-MDT0000 complete [ 5266.403369] LustreError: 163460:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.421591] LustreError: 163460:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5269.591808] LustreError: 163446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773611942 with bad export cookie 12147662316134143548 [ 5269.601322] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.606448] LustreError: 163446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5270.021869] Lustre: server umount lustre-MDT0001 complete [ 5278.274772] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:59:09 (1773611949) [ 5294.809230] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5306.100263] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5342.840153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.142790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5343.183449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5343.261136] Lustre: lustre-MDT0000: new disk, initializing [ 5343.349241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5343.359290] Lustre: Skipped 2 previous similar messages [ 5343.380275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5349.281890] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5359.383951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.566427] Lustre: 169380:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5359.895049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5364.384442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5371.829251] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5375.055891] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5382.580940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5388.460747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5389.633115] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5396.049981] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5404.132769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.140283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5404.142157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5404.142164] Lustre: Skipped 9 previous similar messages [ 5404.165349] Lustre: Skipped 3 previous similar messages [ 5404.200632] LustreError: Skipped 1 previous similar message [ 5408.506904] Lustre: server umount lustre-OST0000 complete [ 5418.732711] Lustre: server umount lustre-MDT0000 complete [ 5419.495056] LustreError: 169385:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.517872] LustreError: 169385:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5422.259733] LustreError: 169372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612094 with bad export cookie 12147662316134144640 [ 5422.263594] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.270744] LustreError: 169372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5422.573491] Lustre: server umount lustre-MDT0001 complete [ 5432.170130] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5434.524939] Key type lgssc unregistered [ 5434.816864] LNet: 172867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5434.825769] LNetError: 172867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5434.845384] LNet: Removed LNI 192.168.203.134@tcp [ 5435.722448] Key type .llcrypt unregistered [ 5435.725435] Key type ._llcrypt unregistered [ 5459.381659] Key type ._llcrypt registered [ 5459.384313] Key type .llcrypt registered [ 5459.483734] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5460.307938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5460.365120] alg: No test for adler32 (adler32-zlib) [ 5461.397256] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 5461.578654] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5463.248257] Key type lgssc registered [ 5464.137275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5471.346337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5471.375372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.885701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.461941] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5482.472233] LustreError: 174268:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5482.486334] LustreError: 174268:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5485.710907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.930914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5489.846092] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5497.210294] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5504.112291] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5513.421434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.738088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.838167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5521.736438] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5529.320791] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5551.589503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.596990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5551.600642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5556.738966] Lustre: server umount lustre-OST0000 complete [ 5561.829226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.846342] Lustre: Skipped 1 previous similar message [ 5561.862575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5561.868846] Lustre: Skipped 2 previous similar messages [ 5566.869871] Lustre: server umount lustre-MDT0000 complete [ 5566.948671] LustreError: 176044:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.970800] LustreError: 176044:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5570.255675] LustreError: 174247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612242 with bad export cookie 10796458103833775885 [ 5570.267744] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5570.649415] Lustre: server umount lustre-MDT0001 complete [ 5582.080730] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5584.802791] Key type lgssc unregistered [ 5585.080778] LNet: 177471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5585.086686] LNetError: 177471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5585.108033] LNet: Removed LNI 192.168.203.134@tcp [ 5585.772562] Key type .llcrypt unregistered [ 5585.776279] Key type ._llcrypt unregistered [ 5608.816333] Key type ._llcrypt registered [ 5608.818319] Key type .llcrypt registered [ 5608.938590] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5610.392184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5610.444094] alg: No test for adler32 (adler32-zlib) [ 5611.631464] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 5611.878173] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5613.528608] Key type lgssc registered [ 5614.474951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5622.424231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5622.444093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.917548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.283172] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5633.510666] LustreError: 178871:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.527832] LustreError: 178871:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5636.249538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.706372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5641.020452] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5647.125256] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5651.296248] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5658.481812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5658.917309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.881880] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5669.194868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5670.259382] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5677.541303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.546208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.560683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.454781] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 5679.588656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.595600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.596534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5683.040988] Lustre: server umount lustre-OST0000 complete [ 5689.415316] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90fd07a5b000.ost_server_uuid 50 [ 5691.249139] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5705.185913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.188361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5705.207532] Lustre: Skipped 1 previous similar message [ 5705.236906] Lustre: Skipped 2 previous similar messages [ 5711.317760] Lustre: server umount lustre-MDT0000 complete [ 5711.345178] LustreError: 178866:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.363050] LustreError: 178866:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5715.025353] LustreError: 178854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612387 with bad export cookie 2304161310453940718 [ 5715.039582] LustreError: 178854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5715.042376] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.596378] Lustre: server umount lustre-MDT0001 complete [ 5723.932175] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:06:35 (1773612395) [ 5730.167591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.501762] LustreError: 182354:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.557140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.684279] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5735.916051] LustreError: 182355:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.025824] LustreError: 182354:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.677586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.085678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5746.932838] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5751.798667] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5755.264849] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5761.128126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.505821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.078280] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5771.944980] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5775.865416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5778.043129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.315601] Lustre: lustre-OST0001: new disk, initializing [ 5778.321427] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5778.415084] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5784.436189] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5785.140075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5785.166900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5785.312672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5791.673396] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5795.827067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.848222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5795.856464] Lustre: Skipped 3 previous similar messages [ 5800.299700] Lustre: server umount lustre-OST0000 complete [ 5800.945361] LustreError: 183948:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.967951] LustreError: 183948:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5806.668715] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5806.850428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5811.402604] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5811.619346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5818.174847] LustreError: 184250:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.198520] LustreError: 184250:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 5826.529993] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5826.530615] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.543834] LustreError: Skipped 1 previous similar message [ 5826.545495] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5826.545503] Lustre: Skipped 1 previous similar message [ 5826.585597] Lustre: Skipped 2 previous similar messages [ 5831.361580] Lustre: server umount lustre-OST0001 complete [ 5836.824791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.438952] Lustre: server umount lustre-MDT0000 complete [ 5846.126786] LustreError: 182336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612518 with bad export cookie 2304161310453941789 [ 5846.136868] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5846.138958] LustreError: 182336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5846.797225] Lustre: server umount lustre-MDT0001 complete [ 5870.883915] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:09:02 (1773612542) [ 5876.689434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.017958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5877.049538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.291756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.841622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5890.589528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5890.734437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5890.772278] Lustre: 188150:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5895.531500] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5901.284403] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5905.271126] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5912.751915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.009919] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5919.742419] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5922.517556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5925.261058] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5931.685348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5931.777725] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5937.898586] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5943.327913] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5948.388996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5948.389319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.393025] LustreError: Skipped 1 previous similar message [ 5948.395180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5948.395189] Lustre: Skipped 3 previous similar messages [ 5948.421982] Lustre: Skipped 2 previous similar messages [ 5952.149822] Lustre: server umount lustre-OST0000 complete [ 5954.375300] LustreError: 190251:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.406775] LustreError: 190251:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 5969.896843] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.927155] Lustre: server umount lustre-OST0001 complete [ 5978.081286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5978.091235] LustreError: Skipped 1 previous similar message [ 5980.642168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.646825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.667570] Lustre: Skipped 3 previous similar messages [ 5980.693671] Lustre: Skipped 9 previous similar messages [ 5983.685558] Lustre: server umount lustre-MDT0000 complete [ 5987.190954] LustreError: 188889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612659 with bad export cookie 2304161310453943196 [ 5987.203582] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.206529] LustreError: 188889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5987.604836] Lustre: server umount lustre-MDT0001 complete [ 6010.037750] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:11:21 (1773612681) [ 6026.901598] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 6036.018056] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6070.357213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.567146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6070.591433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6070.663190] Lustre: lustre-MDT0000: new disk, initializing [ 6070.724278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.730108] Lustre: Skipped 3 previous similar messages [ 6070.744510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6074.737348] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6084.963556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.118538] Lustre: 195757:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6085.160846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6085.164210] Lustre: Skipped 1 previous similar message [ 6085.242180] Lustre: lustre-MDT0001: new disk, initializing [ 6085.359778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6085.378256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6088.826869] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6095.849610] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6099.096529] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6105.354589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6105.589792] Lustre: lustre-OST0000: new disk, initializing [ 6105.593066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6110.834917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6110.852450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6110.964986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6111.903764] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6119.773657] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6124.307265] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6124.596507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6128.846389] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6129.048311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6131.684172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6131.688360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.692815] LustreError: Skipped 1 previous similar message [ 6131.696247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6131.739842] Lustre: Skipped 1 previous similar message [ 6137.045821] Lustre: server umount lustre-OST0000 complete [ 6146.771559] Lustre: server umount lustre-MDT0000 complete [ 6147.050475] LustreError: 195765:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.062124] LustreError: 195765:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 13 previous similar messages [ 6150.330514] LustreError: 197500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612822 with bad export cookie 2304161310453944757 [ 6150.336374] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.344350] LustreError: 197500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6150.716500] Lustre: server umount lustre-MDT0001 complete [ 6159.181379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.760644] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6172.821784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.370351] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6181.910494] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6185.120403] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6191.116732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.334405] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6203.059488] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6207.362593] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6207.525834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6211.595772] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6211.784649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6219.744614] Lustre: server umount lustre-OST0000 complete [ 6225.653175] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6229.858433] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6257.485859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6257.962218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6257.964994] Lustre: Skipped 5 previous similar messages [ 6263.504861] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6269.185485] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6277.392846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6280.431332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.440321] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.443508] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6285.803513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.813869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6285.814748] Lustre: Skipped 5 previous similar messages [ 6285.815061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6285.827189] Lustre: Skipped 8 previous similar messages [ 6285.848424] LustreError: Skipped 1 previous similar message [ 6287.259294] Lustre: server umount lustre-OST0000 complete [ 6296.965695] Lustre: server umount lustre-MDT0000 complete [ 6300.078300] LustreError: 199552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773612972 with bad export cookie 2304161310453945639 [ 6300.084961] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.096837] LustreError: 199552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6300.381326] Lustre: server umount lustre-MDT0001 complete [ 6307.774463] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:16:19 (1773612979) [ 6313.269926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.686249] LustreError: 205023:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.705099] LustreError: 205023:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6317.661053] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6324.787198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.219146] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6332.633776] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6335.582976] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6340.973354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.274565] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6352.086491] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6355.475927] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6359.936380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6364.146137] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6364.424723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6370.772967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6370.947360] Lustre: lustre-OST0001: new disk, initializing [ 6370.951151] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6376.165627] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6379.140740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6379.151374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6379.236865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6382.317781] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6385.940446] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6386.163439] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6389.850079] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6390.033558] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6394.855725] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.860826] LustreError: Skipped 1 previous similar message [ 6397.887932] Lustre: server umount lustre-OST0001 complete [ 6402.816925] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6403.039394] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6406.337193] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6406.470367] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6433.375810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6438.198723] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6438.898221] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6438.916682] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6438.917706] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6438.939685] Lustre: Skipped 2 previous similar messages [ 6442.702385] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6444.057922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6451.853889] Lustre: server umount lustre-OST0001 complete [ 6473.692736] Lustre: server umount lustre-OST0000 complete [ 6482.986289] Lustre: server umount lustre-MDT0000 complete [ 6485.584164] LustreError: 205004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613158 with bad export cookie 2304161310453946598 [ 6485.588930] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6485.596390] LustreError: 205004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6491.920601] Lustre: server umount lustre-MDT0001 complete [ 6511.444301] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:19:43 (1773613183) [ 6516.572384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.901504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6516.938760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6517.078188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.083556] Lustre: Skipped 5 previous similar messages [ 6519.786205] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6526.481513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.565542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6526.588546] Lustre: 213001:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6529.988365] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6534.112877] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6537.083880] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6542.684611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6542.945759] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6548.182868] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6552.896970] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6561.084383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6561.180891] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6566.177341] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6570.112779] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6571.264454] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6574.138620] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6574.263439] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6576.878940] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6576.984639] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6579.338677] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90fd11132000.ost_server_uuid 50 [ 6580.281324] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90fd11132000.ost_server_uuid in FULL state after 0 sec [ 6580.880626] Lustre: Permanently deactivating lustre-OST0001 [ 6580.899724] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6583.054685] Lustre: Permanently reactivating lustre-OST0001 [ 6585.834111] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.839071] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.842167] LustreError: Skipped 4 previous similar messages [ 6585.848289] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6585.858245] Lustre: Skipped 11 previous similar messages [ 6585.886681] Lustre: Skipped 16 previous similar messages [ 6591.877796] Lustre: server umount lustre-OST0001 complete [ 6596.077671] LustreError: 215059:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.092271] LustreError: 215059:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 34 previous similar messages [ 6609.379748] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.482176] Lustre: server umount lustre-OST0000 complete [ 6626.272168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6626.523465] Lustre: server umount lustre-MDT0000 complete [ 6629.311934] LustreError: 214644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613301 with bad export cookie 2304161310453947893 [ 6629.314534] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6629.325157] LustreError: 214644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6629.580926] Lustre: server umount lustre-MDT0001 complete [ 6647.358459] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:21:58 (1773613318) [ 6653.678095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.903617] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6653.959656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6653.966762] Lustre: Skipped 5 previous similar messages [ 6658.004128] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6665.029782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.098825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6665.121886] Lustre: 219104:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6668.963989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6673.328795] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6676.425175] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6682.186692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.363698] Lustre: Permanently deactivating lustre-OST0000 [ 6682.371668] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6686.968788] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6688.866868] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6689.026459] LustreError: 221182:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6695.539700] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6705.197445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6705.325859] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6705.334739] Lustre: Skipped 1 previous similar message [ 6711.536243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6715.147540] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6723.155451] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6725.433646] Lustre: Permanently reactivating lustre-OST0000 [ 6725.468030] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6725.479217] Lustre: Skipped 2 previous similar messages [ 6735.932050] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6735.940701] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6735.955201] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6735.993620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6736.039104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6755.296189] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6755.603372] Lustre: server umount lustre-OST0001 complete [ 6765.648094] Lustre: server umount lustre-OST0000 complete [ 6775.853204] Lustre: server umount lustre-MDT0000 complete [ 6779.510397] LustreError: 219098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613452 with bad export cookie 2304161310453949545 [ 6779.517124] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.526909] LustreError: 219098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6779.868634] Lustre: server umount lustre-MDT0001 complete [ 6790.202467] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6792.602876] Key type lgssc unregistered [ 6792.912770] LNet: 223520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6792.926823] LNetError: 223520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6793.974407] LNet: Removed LNI 192.168.203.134@tcp [ 6794.856154] Key type .llcrypt unregistered [ 6794.862745] Key type ._llcrypt unregistered [ 6814.728323] Key type ._llcrypt registered [ 6814.736791] Key type .llcrypt registered [ 6814.847880] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6816.156278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6816.187899] alg: No test for adler32 (adler32-zlib) [ 6817.315277] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 6817.551904] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 6819.256133] Key type lgssc registered [ 6820.265391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6830.186232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6830.223217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.995344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6836.674985] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6842.340400] LustreError: 224963:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.368825] LustreError: 224963:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6843.367305] LustreError: 224964:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.107227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.455744] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6847.528561] Lustre: 224953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6847.565565] Lustre: Permanently deactivating lustre-MDT0001 [ 6847.616746] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6848.347263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6854.002412] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6855.289810] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6862.837990] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6867.635792] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6874.945599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6875.478785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6877.568981] LustreError: 226578:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.583655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6882.801353] LustreError: 226689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.817251] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6887.910617] LustreError: 226579:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.922299] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6897.042041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.498258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6902.775740] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6902.789596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6902.981198] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6907.189549] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6909.896786] Lustre: Permanently reactivating lustre-MDT0001 [ 6909.911188] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6909.915219] Lustre: Skipped 2 previous similar messages [ 6915.689638] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.706814] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6915.716594] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6915.730728] LustreError: 228084:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6915.731195] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6915.743736] LustreError: 228084:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6915.744047] LustreError: 228084:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -108 [ 6915.744068] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6915.744068] [ 6915.745324] LustreError: 224955:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6916.456901] Lustre: lustre-MDT0001: Client 8960a2bf-9c65-4540-8bf8-88be69b3aab0 (at 192.168.203.34@tcp) reconnecting [ 6920.044124] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6921.838337] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6928.086766] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6928.414801] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6933.176754] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6933.456507] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6935.867103] Lustre: Permanently deactivating lustre-MDT0001 [ 6935.906473] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6935.913744] Lustre: Skipped 2 previous similar messages [ 6946.449584] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6956.005591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6956.012455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.026832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.076861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.077618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.099586] Lustre: Skipped 2 previous similar messages [ 6959.114825] Lustre: Skipped 3 previous similar messages [ 6961.240529] Lustre: server umount lustre-MDT0000 complete [ 6964.194964] LustreError: 224964:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.217866] LustreError: 224964:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [ 6965.059924] LustreError: 224943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613637 with bad export cookie 1343761560180566221 [ 6965.063946] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6965.068050] LustreError: 224943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6965.416832] Lustre: server umount lustre-MDT0001 complete [ 6974.617924] Lustre: server umount lustre-OST0000 complete [ 6985.522666] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:27:36 (1773613656) [ 6995.688607] Lustre: server umount lustre-OST0001 complete [ 7002.313633] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 7011.913929] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7044.101610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.580948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7044.591115] Lustre: Skipped 2 previous similar messages [ 7044.611951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7044.713376] Lustre: lustre-MDT0000: new disk, initializing [ 7044.777261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7044.790801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7049.164379] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7060.036607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.201505] Lustre: 232658:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7060.293481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7060.297266] Lustre: Skipped 1 previous similar message [ 7060.420662] Lustre: lustre-MDT0001: new disk, initializing [ 7060.507736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7060.551956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7060.562078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7065.653930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7075.482814] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7079.937742] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7086.942121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.197650] Lustre: lustre-OST0000: new disk, initializing [ 7087.202652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7087.281816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.591916] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7096.371709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7096.389068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7096.474379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7101.621765] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7103.703891] LustreError: 232665:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7112.514474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7112.727750] Lustre: lustre-OST0001: new disk, initializing [ 7112.732356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7113.794138] LustreError: 232665:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7113.843249] LustreError: 232666:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7118.493867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7118.516515] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7118.662229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7119.671476] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7123.860374] LustreError: 232666:0:(mdt_handler.c:3251:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7129.581914] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7131.855946] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7134.182767] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7134.187863] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.192452] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7134.194226] LustreError: Skipped 1 previous similar message [ 7134.214725] Lustre: Skipped 1 previous similar message [ 7139.297443] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7139.320958] Lustre: Skipped 2 previous similar messages [ 7140.270436] Lustre: server umount lustre-OST0001 complete [ 7144.419558] LustreError: 234387:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.433930] LustreError: 234387:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7158.241392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7158.242473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7158.261924] LustreError: Skipped 1 previous similar message [ 7158.262609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7158.292024] Lustre: Skipped 1 previous similar message [ 7163.528174] Lustre: server umount lustre-OST0000 complete [ 7167.969412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7167.982317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7168.005326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7168.015277] Lustre: Skipped 3 previous similar messages [ 7173.763226] Lustre: server umount lustre-MDT0000 complete [ 7177.715756] LustreError: 232650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773613850 with bad export cookie 1343761560180568818 [ 7177.719390] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7177.739510] LustreError: 232650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7184.418937] Lustre: server umount lustre-MDT0001 complete [ 7197.184400] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7201.176392] Key type lgssc unregistered [ 7201.450749] LNet: 237307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7201.463627] LNetError: 237307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7201.495625] LNet: Removed LNI 192.168.203.134@tcp [ 7202.385759] Key type .llcrypt unregistered [ 7202.388941] Key type ._llcrypt unregistered [ 7246.414939] Key type ._llcrypt registered [ 7246.419417] Key type .llcrypt registered [ 7246.569622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7248.018674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7248.210415] alg: No test for adler32 (adler32-zlib) [ 7249.438355] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7249.709231] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7251.464464] Key type lgssc registered [ 7252.753929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7261.033957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7261.058805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.360821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7262.379951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7262.637716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.636266] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7277.708190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.858773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7277.903669] Lustre: 239499:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7278.245567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7282.733385] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7287.841437] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7291.620480] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7299.439686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7299.714751] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7299.976903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.421739] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7310.020851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7312.608735] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7366.626285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7366.637068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.648621] LustreError: Skipped 1 previous similar message [ 7366.655994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.684185] Lustre: Skipped 1 previous similar message [ 7366.714388] Lustre: Skipped 1 previous similar message [ 7371.751526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.772756] Lustre: Skipped 1 previous similar message [ 7372.076302] Lustre: server umount lustre-OST0000 complete [ 7376.811446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7390.977867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7391.255924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.077540] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7397.414779] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7397.437313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7397.440828] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7397.457352] Lustre: Skipped 1 previous similar message [ 7402.967689] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7448.551157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7448.554815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.569031] LustreError: Skipped 1 previous similar message [ 7448.572251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7448.578847] Lustre: Skipped 1 previous similar message [ 7448.592647] Lustre: Skipped 1 previous similar message [ 7449.844708] Lustre: server umount lustre-OST0000 complete [ 7453.667261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.668675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.674141] Lustre: Skipped 1 previous similar message [ 7453.684981] Lustre: Skipped 1 previous similar message [ 7458.787159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7458.790103] Lustre: Skipped 1 previous similar message [ 7459.904421] Lustre: server umount lustre-MDT0000 complete [ 7464.531453] LustreError: 239489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614137 with bad export cookie 9307223290384569369 [ 7464.551765] LustreError: 239489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7464.565212] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7464.851934] Lustre: server umount lustre-MDT0001 complete [ 7475.896757] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7478.453940] Key type lgssc unregistered [ 7478.680629] LNet: 244458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7478.684465] LNetError: 244458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7478.704430] LNet: Removed LNI 192.168.203.134@tcp [ 7479.362309] Key type .llcrypt unregistered [ 7479.364845] Key type ._llcrypt unregistered [ 7507.296194] Key type ._llcrypt registered [ 7507.300752] Key type .llcrypt registered [ 7507.431622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7509.423225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7509.471771] alg: No test for adler32 (adler32-zlib) [ 7510.652681] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7510.975684] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7512.704267] Key type lgssc registered [ 7513.881083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7523.011486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7523.050581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.772075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.088092] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7535.075167] LustreError: 245856:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.105781] LustreError: 245856:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7537.952898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.295778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7542.922858] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7548.580790] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7552.555287] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7559.519604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.969296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.089559] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7570.294445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7573.571529] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7599.074862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7599.077847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.089415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.135550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.139484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.145460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7603.368261] Lustre: server umount lustre-OST0000 complete [ 7609.828422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7609.841121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.863499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7609.873334] Lustre: Skipped 1 previous similar message [ 7613.621293] Lustre: server umount lustre-MDT0000 complete [ 7616.486194] LustreError: 245856:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7616.503271] LustreError: 245856:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7617.708292] LustreError: 245837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614290 with bad export cookie 8862878875804055702 [ 7617.708887] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7617.724089] LustreError: 245837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7618.101704] Lustre: server umount lustre-MDT0001 complete [ 7627.576512] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7630.765384] Key type lgssc unregistered [ 7631.086831] LNet: 249805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7631.098437] LNetError: 249805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7632.175131] LNet: Removed LNI 192.168.203.134@tcp [ 7633.174541] Key type .llcrypt unregistered [ 7633.181467] Key type ._llcrypt unregistered [ 7655.615457] Key type ._llcrypt registered [ 7655.617448] Key type .llcrypt registered [ 7655.771994] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7657.240772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.628209] alg: No test for adler32 (adler32-zlib) [ 7658.710179] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7659.057869] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7660.816548] Key type lgssc registered [ 7662.285396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7672.053587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7672.105201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.946211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.591869] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7684.066762] LustreError: 251013:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.093860] LustreError: 251013:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7689.207365] LustreError: 251012:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.491846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.891826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.826617] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7701.012535] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7705.687415] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7714.578962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7715.066749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.202996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7721.739282] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7728.722050] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7741.921772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7741.938036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.963355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7746.017137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7746.036479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7746.048711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7748.176782] Lustre: server umount lustre-OST0000 complete [ 7756.257223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7756.258518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.270944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.279977] Lustre: Skipped 1 previous similar message [ 7756.292165] Lustre: Skipped 2 previous similar messages [ 7758.393717] Lustre: server umount lustre-MDT0000 complete [ 7761.380523] LustreError: 251013:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.722391] LustreError: 253316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614434 with bad export cookie 11279118064517815116 [ 7761.724761] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7761.730122] LustreError: 253316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7762.133334] Lustre: server umount lustre-MDT0001 complete [ 7772.278486] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7775.408257] Key type lgssc unregistered [ 7775.729675] LNet: 254339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7775.733234] LNetError: 254339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7775.746886] LNet: Removed LNI 192.168.203.134@tcp [ 7776.463934] Key type .llcrypt unregistered [ 7776.466920] Key type ._llcrypt unregistered [ 7804.902227] Key type ._llcrypt registered [ 7804.903794] Key type .llcrypt registered [ 7805.100780] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7806.364569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.570360] alg: No test for adler32 (adler32-zlib) [ 7807.687826] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7808.068260] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7809.762095] Key type lgssc registered [ 7810.994277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7819.369781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7819.387836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.871349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.490716] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7831.011379] LustreError: 255739:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.027842] LustreError: 255739:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7834.175295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.574099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.339187] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7845.321923] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7850.012540] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7858.271268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.590141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.167275] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7865.779151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7871.422571] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7900.641749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7900.652230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.690830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7901.672474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7901.689229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.708634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7905.116421] Lustre: server umount lustre-OST0000 complete [ 7911.394893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7911.414862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.431563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7911.437502] Lustre: Skipped 1 previous similar message [ 7916.511437] Lustre: server umount lustre-MDT0000 complete [ 7917.033147] LustreError: 255733:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7917.048733] LustreError: 255733:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7922.064060] LustreError: 255720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614594 with bad export cookie 13863807090217679358 [ 7922.071684] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.080318] LustreError: 255720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7922.719288] Lustre: server umount lustre-MDT0001 complete [ 7935.914971] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 7939.114518] Key type lgssc unregistered [ 7939.545550] LNet: 259691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7939.562240] LNetError: 259691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7939.580731] LNet: Removed LNI 192.168.203.134@tcp [ 7940.426135] Key type .llcrypt unregistered [ 7940.429496] Key type ._llcrypt unregistered [ 7961.744564] Key type ._llcrypt registered [ 7961.749499] Key type .llcrypt registered [ 7961.903845] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7963.499947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7963.556577] alg: No test for adler32 (adler32-zlib) [ 7964.689901] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 7964.948821] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 7966.632194] Key type lgssc registered [ 7967.960441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7976.889373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7976.934084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.608993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.820255] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7988.708051] LustreError: 260913:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7988.728169] LustreError: 260913:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7993.737654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.828086] LustreError: 260915:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7994.098823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7998.826614] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8005.092511] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8009.477960] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8016.656975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8017.134806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8024.013823] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8024.332753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8031.078955] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8045.033086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8045.048735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.054186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8045.072614] Lustre: Skipped 1 previous similar message [ 8049.392858] Lustre: server umount lustre-OST0000 complete [ 8055.266095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.296031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.298809] Lustre: Skipped 1 previous similar message [ 8059.848418] Lustre: server umount lustre-MDT0000 complete [ 8060.386125] LustreError: 260921:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8060.407887] LustreError: 260921:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8063.672624] LustreError: 263528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614736 with bad export cookie 10488677808748175711 [ 8063.681091] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.684940] LustreError: 263528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8064.013722] Lustre: server umount lustre-MDT0001 complete [ 8075.035628] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8078.464499] Key type lgssc unregistered [ 8078.922429] LNet: 264262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8078.928754] LNetError: 264262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8078.940269] LNet: Removed LNI 192.168.203.134@tcp [ 8079.878559] Key type .llcrypt unregistered [ 8079.881876] Key type ._llcrypt unregistered [ 8093.406760] Key type ._llcrypt registered [ 8093.418167] Key type .llcrypt registered [ 8093.548486] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8111.122815] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8127.760579] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8129.614401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8129.921905] alg: No test for adler32 (adler32-zlib) [ 8131.147313] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8131.480594] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8133.344193] Key type lgssc registered [ 8134.562922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8170.103159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8170.150279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.647937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8171.713087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8171.834170] Lustre: lustre-MDT0000: new disk, initializing [ 8171.978158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.008423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8176.980974] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8188.727800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.952770] Lustre: 267767:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8189.075134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8189.077698] Lustre: Skipped 1 previous similar message [ 8189.207629] Lustre: lustre-MDT0001: new disk, initializing [ 8189.377124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.415728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8189.425915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8193.700855] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8201.311826] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8204.767683] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8211.544324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8211.799903] Lustre: lustre-OST0000: new disk, initializing [ 8211.804928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8211.917875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8218.227364] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8218.716086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8218.734025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8218.893639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8226.238343] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8231.250233] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8231.594365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8236.637270] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8236.885249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8239.589952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8239.592640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.599573] LustreError: Skipped 1 previous similar message [ 8239.601303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.624143] Lustre: Skipped 1 previous similar message [ 8244.705608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8244.707307] Lustre: Skipped 1 previous similar message [ 8249.839841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8249.859918] Lustre: Skipped 1 previous similar message [ 8252.896442] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8253.049496] Lustre: server umount lustre-OST0000 complete [ 8260.071683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8260.089407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8260.089715] Lustre: Skipped 1 previous similar message [ 8260.098510] Lustre: Skipped 1 previous similar message [ 8263.107423] Lustre: server umount lustre-MDT0000 complete [ 8267.230857] LustreError: 267761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773614939 with bad export cookie 6229818937634875226 [ 8267.238603] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8267.244830] LustreError: 267761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8267.950655] Lustre: server umount lustre-MDT0001 complete [ 8278.375079] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:49:09 (1773614949) [ 8285.306517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.818479] LustreError: 271776:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8285.836160] LustreError: 271776:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 8285.886821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8289.671789] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8291.299487] LustreError: 271777:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.419757] LustreError: 271776:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.340931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.698493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8303.112468] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8308.884255] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8312.628855] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8319.757539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8320.087130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.639578] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8332.602642] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8341.998433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.015354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.037727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8342.040356] Lustre: Skipped 1 previous similar message [ 8343.634312] Lustre: server umount lustre-OST0000 complete [ 8349.768822] Lustre: server umount lustre-MDT0000 complete [ 8350.177205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8350.193971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.207603] Lustre: Skipped 1 previous similar message [ 8350.212714] LustreError: 271776:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8353.750514] LustreError: 271756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615026 with bad export cookie 6229818937634876108 [ 8353.770090] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.774108] LustreError: 271756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8354.235725] Lustre: server umount lustre-MDT0001 complete [ 8366.206037] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8369.344344] Key type lgssc unregistered [ 8369.839633] LNet: 274958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8369.867500] LNetError: 274958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8369.886752] LNet: Removed LNI 192.168.203.134@tcp [ 8370.782171] Key type .llcrypt unregistered [ 8370.786145] Key type ._llcrypt unregistered [ 8398.614993] Key type ._llcrypt registered [ 8398.621510] Key type .llcrypt registered [ 8398.750512] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8414.038363] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8415.585624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8415.738968] alg: No test for adler32 (adler32-zlib) [ 8416.906714] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8417.201763] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8418.896234] Key type lgssc registered [ 8420.323646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8471.639786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8471.708453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.999724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8473.063242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8473.168299] Lustre: lustre-MDT0000: new disk, initializing [ 8473.282081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.306638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8477.752246] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8488.601593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.774146] Lustre: 278751:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8488.881782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8488.891651] Lustre: Skipped 1 previous similar message [ 8488.987880] Lustre: lustre-MDT0001: new disk, initializing [ 8489.072740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8489.095568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8489.109179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8494.814850] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8503.552202] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8509.251408] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8518.931726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8519.293656] Lustre: lustre-OST2710: new disk, initializing [ 8519.296632] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8519.393882] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8525.762446] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8528.973572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8528.992550] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8529.161117] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8532.867921] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8540.406829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.587392] Lustre: lustre-OST03e8: new disk, initializing [ 8540.590791] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8540.686094] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8547.215883] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8549.489173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8549.495819] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8549.536280] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8553.416107] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8561.339115] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8561.666249] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8566.429547] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8566.620672] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8572.721719] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8573.029629] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8578.690134] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8579.022841] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8584.478585] Lustre: server umount lustre-MDT0000 complete [ 8585.704950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8585.732457] Lustre: Skipped 1 previous similar message [ 8588.136985] LustreError: 280506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615260 with bad export cookie 15555300875885255683 [ 8588.137880] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8588.149940] LustreError: 280506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8588.910203] Lustre: server umount lustre-MDT0001 complete [ 8603.535522] Lustre: server umount lustre-OST2710 complete [ 8607.202382] Lustre: 276546:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773615263/real 1773615263] req@000000009740c584 x1859770215631488/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773615279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8607.225780] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.260732] Lustre: Skipped 2 previous similar messages [ 8608.300306] Lustre: server umount lustre-OST03e8 complete [ 8630.168087] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 8645.407298] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8687.722764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.226362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8688.285894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8688.460608] Lustre: lustre-MDT0000: new disk, initializing [ 8688.585534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8688.627824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8694.422492] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8706.176199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.313324] Lustre: 286505:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8706.480638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8706.494524] Lustre: Skipped 1 previous similar message [ 8706.688921] Lustre: lustre-MDT0001: new disk, initializing [ 8706.913145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8706.981725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8706.997443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8712.975751] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8721.549727] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8724.933708] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8731.319211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.560813] Lustre: lustre-OST0000: new disk, initializing [ 8731.564527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8731.702804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.929922] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8738.941264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8738.968538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8739.117692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8746.423253] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8751.482414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8751.677976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8756.510581] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8756.734650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8759.777271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8759.788187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8759.791578] LustreError: Skipped 1 previous similar message [ 8759.792314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8759.805979] Lustre: Skipped 1 previous similar message [ 8759.820828] Lustre: Skipped 1 previous similar message [ 8764.897933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8764.916960] Lustre: Skipped 1 previous similar message [ 8770.025633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8770.034084] Lustre: Skipped 1 previous similar message [ 8773.088130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8773.536730] Lustre: server umount lustre-OST0000 complete [ 8778.721616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8778.726942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8778.759303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8783.798319] Lustre: server umount lustre-MDT0000 complete [ 8785.385394] LustreError: 286514:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.409796] LustreError: 286514:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 8788.306209] LustreError: 286497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615460 with bad export cookie 15555300875885257363 [ 8788.317743] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8788.319045] LustreError: 286497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8788.739682] Lustre: server umount lustre-MDT0001 complete [ 8796.982515] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:57:48 (1773615468) [ 8798.681921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8800.549605] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:57:51 (1773615471) [ 8804.142365] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8832.161725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.570819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8832.600393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8832.777196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8837.067993] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8847.974647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.254943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8848.351533] Lustre: 291941:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8856.292782] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 8864.477474] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8869.559495] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8889.465402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8889.647884] Lustre: Denying initial registration attempt from nid 192.168.203.134@tcp, specified as failover [ 8889.657147] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8889.663707] LustreError: 293556:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8889.671252] LustreError: 293556:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 8889.680453] LustreError: 293556:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 8889.765713] Lustre: server umount lustre-OST0000 complete [ 8889.772815] LustreError: 293556:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8900.067083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8900.073549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.081478] Lustre: Skipped 1 previous similar message [ 8900.089897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.092681] Lustre: Skipped 2 previous similar messages [ 8902.402832] Lustre: server umount lustre-MDT0000 complete [ 8905.698099] LustreError: 291947:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.720672] LustreError: 291947:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8906.152081] LustreError: 291933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615578 with bad export cookie 15555300875885258245 [ 8906.157710] LustreError: 291933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8906.161466] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8906.661362] Lustre: server umount lustre-MDT0001 complete [ 8918.002271] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8920.957621] Key type lgssc unregistered [ 8921.327103] LNet: 294661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8921.337700] LNetError: 294661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8921.361741] LNet: Removed LNI 192.168.203.134@tcp [ 8922.060502] Key type .llcrypt unregistered [ 8922.062733] Key type ._llcrypt unregistered [ 8934.716724] Key type ._llcrypt registered [ 8934.727459] Key type .llcrypt registered [ 8934.885430] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 8936.973699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8937.156231] alg: No test for adler32 (adler32-zlib) [ 8938.611809] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 8938.948089] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 8940.747679] Key type lgssc registered [ 8942.522852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9001.391363] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9010.388967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9010.420493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.987461] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9012.051830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9012.558104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.954570] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9027.762260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.884987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9027.924350] Lustre: 297414:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9028.397802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9033.282128] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9040.804859] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9045.738589] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9066.867807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9067.195555] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9067.580595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9075.211751] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9082.072406] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9086.433677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9086.434742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.450135] LustreError: Skipped 1 previous similar message [ 9086.456265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9086.504140] Lustre: Skipped 1 previous similar message [ 9090.649365] Lustre: server umount lustre-OST0000 complete [ 9095.139498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9095.156196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.186984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9095.196879] Lustre: Skipped 1 previous similar message [ 9096.675217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.682384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9101.349195] Lustre: server umount lustre-MDT0000 complete [ 9106.178077] LustreError: 297405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615778 with bad export cookie 1989238455259950258 [ 9106.198976] LustreError: 297405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9106.201219] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9106.918841] Lustre: server umount lustre-MDT0001 complete [ 9119.036650] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9122.074525] Key type lgssc unregistered [ 9122.280291] LNet: 300631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9122.285878] LNetError: 300631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9122.304555] LNet: Removed LNI 192.168.203.134@tcp [ 9123.106575] Key type .llcrypt unregistered [ 9123.108527] Key type ._llcrypt unregistered [ 9147.578602] Key type ._llcrypt registered [ 9147.582154] Key type .llcrypt registered [ 9147.724930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9149.041424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9149.085369] alg: No test for adler32 (adler32-zlib) [ 9150.204654] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 9150.443438] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9152.145052] Key type lgssc registered [ 9153.279744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9160.271197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9160.296524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.040918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9166.854698] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9172.458088] LustreError: 302031:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9172.496207] LustreError: 302031:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9177.327659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.594078] LustreError: 302032:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9177.885039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9183.481877] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9190.560478] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9194.441540] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9203.018452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9203.396327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.880643] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9216.150115] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9233.382972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.395114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9233.399894] Lustre: Skipped 1 previous similar message [ 9233.410090] Lustre: Skipped 2 previous similar messages [ 9234.280326] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 9236.987326] Lustre: server umount lustre-MDT0000 complete [ 9238.498834] LustreError: 303239:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.515695] LustreError: 303239:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9240.825777] LustreError: 304302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773615913 with bad export cookie 1222527069851470507 [ 9240.830258] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.835159] LustreError: 304302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9243.617492] LustreError: 303238:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.618235] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9243.650236] LustreError: 303238:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9243.686763] Lustre: Skipped 1 previous similar message [ 9243.690497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9249.604956] LustreError: 302027:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.615115] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 9249.637606] LustreError: 302027:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9249.658316] Lustre: Skipped 2 previous similar messages [ 9254.723062] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 9254.734033] Lustre: Skipped 1 previous similar message [ 9255.392598] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9255.778938] Lustre: server umount lustre-MDT0001 complete [ 9261.985896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9270.625342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.601409] LustreError: 301341:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000289472aa x1859770984214528/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9286.121486] LustreError: 303638:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.144731] LustreError: 303638:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9286.268233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9286.350843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9291.511711] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9305.102815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.739084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9306.732236] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9313.444599] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9320.257060] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9322.026472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9325.668177] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9327.463548] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9332.710761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.736978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9337.316784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9337.344615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.754873] Lustre: server umount lustre-OST0000 complete [ 9342.955309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.993300] Lustre: Skipped 1 previous similar message [ 9353.192545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9353.200122] Lustre: Skipped 7 previous similar messages [ 9357.280317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9357.526349] Lustre: server umount lustre-MDT0000 complete [ 9358.309375] LustreError: 305319:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.346368] LustreError: 305319:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9361.687509] LustreError: 302012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616034 with bad export cookie 1222527069851478354 [ 9361.703167] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.716714] LustreError: 302012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9362.282650] Lustre: server umount lustre-MDT0001 complete [ 9373.745950] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9376.917337] Key type lgssc unregistered [ 9377.142935] LNet: 307752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9377.147347] LNetError: 307752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9377.160711] LNet: Removed LNI 192.168.203.134@tcp [ 9377.981873] Key type .llcrypt unregistered [ 9377.984295] Key type ._llcrypt unregistered [ 9408.727842] Key type ._llcrypt registered [ 9408.729849] Key type .llcrypt registered [ 9408.833479] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9410.236185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9410.542378] alg: No test for adler32 (adler32-zlib) [ 9411.722868] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 9411.963885] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9413.745647] Key type lgssc registered [ 9415.558353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9425.850838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9425.866831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9427.480633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9431.266478] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9436.646147] LustreError: 309154:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9436.664281] LustreError: 309154:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9440.270861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.769472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9445.418719] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9450.592761] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9454.564227] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9461.221724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9461.245344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9461.290110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9462.242388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.244874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9462.271610] Lustre: Skipped 1 previous similar message [ 9464.019165] Lustre: server umount lustre-MDT0000 complete [ 9466.404389] LustreError: 309149:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9466.446060] LustreError: 309149:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9468.047107] LustreError: 309134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616140 with bad export cookie 16587846733768684197 [ 9468.067518] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9468.408604] Lustre: server umount lustre-MDT0001 complete [ 9484.386713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.918204] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9484.975070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9485.250384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.289222] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9502.028546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9502.263275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9502.340978] Lustre: 311515:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9502.718722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.607906] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9513.556795] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9519.043217] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9531.968037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9532.538666] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9532.554054] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9532.562752] LustreError: 313176:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9532.567720] LustreError: 313176:0:(tgt_mount.c:2064:server_put_super()) no obd lustre-OST0000 [ 9532.577833] LustreError: 313176:0:(tgt_mount.c:115:server_deregister_mount()) lustre-OST0000 not registered [ 9532.690215] Lustre: server umount lustre-OST0000 complete [ 9532.692293] LustreError: 313176:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9545.258489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9545.530733] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9545.821027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9553.648965] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9555.642045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9559.612591] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9567.431562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9567.559187] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9567.645893] Lustre: lustre-OST0001: new disk, initializing [ 9567.653810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9567.809160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9573.615300] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9576.582370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9576.589883] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9576.693870] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9579.995971] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9585.347970] Lustre: server umount lustre-OST0001 complete [ 9587.173563] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9587.180649] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.206518] LustreError: 313644:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9587.230526] LustreError: 313644:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9591.274286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9591.292698] LustreError: Skipped 1 previous similar message [ 9591.296607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.314047] Lustre: Skipped 1 previous similar message [ 9591.327628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9592.308952] LustreError: 313644:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.329557] LustreError: 313644:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9597.416397] LustreError: 313641:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9597.479120] LustreError: 313641:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9601.596939] Lustre: server umount lustre-OST0000 complete [ 9607.659278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.672497] Lustre: Skipped 1 previous similar message [ 9607.687721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9607.702606] Lustre: Skipped 4 previous similar messages [ 9612.800267] LustreError: 313666:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.278829] Lustre: server umount lustre-MDT0000 complete [ 9618.505934] LustreError: 311510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616291 with bad export cookie 16587846733768684750 [ 9618.507088] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9618.512483] LustreError: 311510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9618.805994] Lustre: server umount lustre-MDT0001 complete [ 9629.098206] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 9632.094850] Key type lgssc unregistered [ 9632.346782] LNet: 316433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9632.363927] LNetError: 316433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9633.388164] LNet: Removed LNI 192.168.203.134@tcp [ 9634.401947] Key type .llcrypt unregistered [ 9634.404922] Key type ._llcrypt unregistered [ 9708.941126] Key type ._llcrypt registered [ 9708.942672] Key type .llcrypt registered [ 9709.013097] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 9728.241810] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 9730.796022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9730.850249] alg: No test for adler32 (adler32-zlib) [ 9731.974449] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [ 9732.469490] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 9734.359991] Key type lgssc registered [ 9736.559908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9780.458992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9780.498450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.914403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9781.979134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9782.124062] Lustre: lustre-MDT0000: new disk, initializing [ 9782.306824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9782.325862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9787.159305] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9799.986919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9800.112498] Lustre: 321335:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9800.199386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9800.203656] Lustre: Skipped 1 previous similar message [ 9800.279918] Lustre: lustre-MDT0001: new disk, initializing [ 9800.385276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9800.419119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9800.429688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9804.496620] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9811.974565] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9815.943985] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9822.470997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.809388] Lustre: lustre-OST0000: new disk, initializing [ 9822.812694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9822.938408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.456336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9828.476383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9828.618390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9828.847290] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9836.457949] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9840.644878] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9840.938845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9845.350725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9845.544245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9849.324212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9849.325677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.329209] LustreError: Skipped 1 previous similar message [ 9849.331394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9849.343843] Lustre: Skipped 1 previous similar message [ 9849.353264] Lustre: Skipped 1 previous similar message [ 9853.324861] Lustre: server umount lustre-OST0000 complete [ 9859.554666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9859.557068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9859.576981] Lustre: Skipped 1 previous similar message [ 9863.625616] Lustre: server umount lustre-MDT0000 complete [ 9866.668495] LustreError: 321328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616539 with bad export cookie 11565408339947720241 [ 9866.668494] LustreError: 322124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616539 with bad export cookie 11565408339947720241 [ 9866.668947] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9866.679228] LustreError: 322124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9866.945032] Lustre: server umount lustre-MDT0001 complete [ 9874.325193] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:15:45 (1773616545) [ 9881.878353] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:15:53 (1773616553) [ 9890.094048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.441180] LustreError: 325685:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.462748] LustreError: 325685:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9890.532433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.270107] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9895.910414] LustreError: 325686:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9901.031535] LustreError: 325685:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.086656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.380317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9906.527728] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9911.273169] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9914.388876] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9919.865866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.138828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9925.380639] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9931.123798] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9943.219764] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9944.611080] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9945.805591] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9947.404970] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9949.317700] Lustre: Failing over lustre-MDT0000 [ 9949.582344] Lustre: server umount lustre-MDT0000 complete [ 9951.204302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.224095] LustreError: 325685:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9951.231098] Lustre: Skipped 2 previous similar messages [ 9951.264772] LustreError: 325685:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 9956.324865] LustreError: 325680:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.350108] LustreError: 325680:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9966.560516] Lustre: 319559:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773616623/real 1773616623] req@00000000e2d16228 x1859771594507136/t0(0) o400->MGC192.168.203.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773616639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9966.584871] LustreError: 327297:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9966.606858] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9966.638423] LustreError: 327297:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 9968.629033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9977.847446] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa080959be8ec95bd [ 9977.868070] Lustre: MGC192.168.203.134@tcp: Connection restored to 0@lo (at 0@lo) [ 9978.213894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9978.282150] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9978.687855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9982.494442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 9983.480213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9983.519591] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9983.560588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9990.460541] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9992.125679] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9995.335345] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9997.114680] Lustre: Failing over lustre-MDT0000 [ 9997.386636] Lustre: server umount lustre-MDT0000 complete [ 9998.821127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9998.826979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9998.835300] LustreError: 328008:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.835310] LustreError: 328008:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 11 previous similar messages [ 9998.888512] Lustre: Skipped 1 previous similar message [10015.203847] Lustre: 319559:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773616671/real 1773616671] req@00000000283dd9f5 x1859771594528256/t0(0) o400->MGC192.168.203.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773616687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10015.239673] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10015.389038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.443330] LustreError: 319556:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002e66e8c1 x1859771594534016/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10025.765263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.813939] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10029.341788] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10029.892867] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10031.104662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10031.122389] Lustre: Skipped 2 previous similar messages [10031.168284] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10031.236290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10036.840640] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10038.336140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10046.437426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.437702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10046.449725] Lustre: Skipped 1 previous similar message [10046.470752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10046.477585] Lustre: Skipped 3 previous similar messages [10050.157616] Lustre: server umount lustre-MDT0000 complete [10051.563375] LustreError: 328995:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.594916] LustreError: 328995:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 28 previous similar messages [10053.365084] LustreError: 325666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616725 with bad export cookie 11565408339947722971 [10053.369206] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.373785] LustreError: 325666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10053.719986] Lustre: server umount lustre-MDT0001 complete [10067.545485] Lustre: server umount lustre-OST0000 complete [10082.535679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.272099] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10096.168887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10096.519912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10096.525024] Lustre: Skipped 1 previous similar message [10100.350734] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10104.777627] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10108.083376] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10114.496047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.209597] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10125.611813] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10128.120490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10137.571140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10137.588386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10137.607387] Lustre: Skipped 2 previous similar messages [10137.616584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10141.931440] Lustre: server umount lustre-OST0000 complete [10147.810082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10147.822493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10147.828362] Lustre: Skipped 2 previous similar messages [10151.226950] Lustre: server umount lustre-MDT0000 complete [10153.958496] LustreError: 333429:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.983502] LustreError: 333429:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [10155.324424] LustreError: 331477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616827 with bad export cookie 11565408339947723713 [10155.330532] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10155.333153] LustreError: 331477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10155.703473] Lustre: server umount lustre-MDT0001 complete [10165.150085] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10167.244763] Key type lgssc unregistered [10167.490786] LNet: 334675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10167.503285] LNetError: 334675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10167.517892] LNet: Removed LNI 192.168.203.134@tcp [10168.106254] Key type .llcrypt unregistered [10168.108463] Key type ._llcrypt unregistered [10188.338480] Key type ._llcrypt registered [10188.343592] Key type .llcrypt registered [10188.425396] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10189.154539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10189.222606] alg: No test for adler32 (adler32-zlib) [10190.267725] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [10190.470729] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10192.112232] Key type lgssc registered [10193.022302] Lustre: Echo OBD driver; http://www.lustre.org/ [10200.356436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10200.376200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10201.968385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10205.775403] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10212.330205] LustreError: 336152:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.346678] LustreError: 336152:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10212.821703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.076020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10215.968655] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10220.340360] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10222.952753] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10228.554354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10228.796503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10233.593677] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10238.284225] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10247.276510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10257.892703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10257.898477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10257.907815] Lustre: Skipped 2 previous similar messages [10257.926667] Lustre: Skipped 2 previous similar messages [10262.964738] Lustre: server umount lustre-MDT0000 complete [10263.012577] LustreError: 336152:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.033685] LustreError: 336152:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10265.898127] LustreError: 336133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773616938 with bad export cookie 10585560967639935183 [10265.902376] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10265.908313] LustreError: 336133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10266.150537] Lustre: server umount lustre-MDT0001 complete [10279.647549] Lustre: server umount lustre-OST0000 complete [10293.032775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.483371] LustreError: 339547:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10293.582914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10296.811083] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10298.851208] LustreError: 339548:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.971129] LustreError: 339547:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.177171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10304.504425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10307.773873] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10311.989137] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10315.238510] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10320.950808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10321.216164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10325.913105] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10329.902755] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10338.616339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10354.151315] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.159767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.166737] Lustre: Skipped 2 previous similar messages [10354.179655] Lustre: Skipped 2 previous similar messages [10359.266943] LustreError: 339548:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.280276] LustreError: 339548:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10359.327351] Lustre: server umount lustre-MDT0000 complete [10362.361617] LustreError: 339528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617034 with bad export cookie 10585560967639936296 [10362.370192] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10362.374640] LustreError: 339528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10362.627697] Lustre: server umount lustre-MDT0001 complete [10376.212425] Lustre: server umount lustre-OST0000 complete [10389.241606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10389.592357] LustreError: 343143:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.606405] LustreError: 343143:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10389.648877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10392.976618] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10400.632073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10404.464599] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10409.001469] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10412.153698] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10417.407910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10422.194219] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10426.374761] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10435.117863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10441.186902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10441.193907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10441.207578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10445.281247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10445.285337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10445.294129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.319820] Lustre: Skipped 1 previous similar message [10446.914162] Lustre: server umount lustre-OST0000 complete [10450.403283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.405777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10450.413358] Lustre: Skipped 1 previous similar message [10450.427254] Lustre: Skipped 1 previous similar message [10456.605621] Lustre: server umount lustre-MDT0000 complete [10459.997056] LustreError: 343124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617132 with bad export cookie 10585560967639937318 [10460.005186] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10460.026036] LustreError: 343124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10460.269200] Lustre: server umount lustre-MDT0001 complete [10468.628571] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10470.732601] Key type lgssc unregistered [10470.980965] LNet: 346342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10470.993914] LNetError: 346342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10472.049260] LNet: Removed LNI 192.168.203.134@tcp [10472.771414] Key type .llcrypt unregistered [10472.773601] Key type ._llcrypt unregistered [10492.322574] Key type ._llcrypt registered [10492.324668] Key type .llcrypt registered [10492.426891] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10493.297656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10493.380688] alg: No test for adler32 (adler32-zlib) [10494.492817] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [10494.686175] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10496.320454] Key type lgssc registered [10497.621179] Lustre: Echo OBD driver; http://www.lustre.org/ [10503.813122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10503.831970] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.836969] LustreError: 347744:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10503.854509] LustreError: 347744:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [10503.860315] LustreError: 347744:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10503.867445] LustreError: 347744:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10503.877566] LustreError: 347744:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [10509.900138] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.905582] LustreError: 348140:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10509.919802] LustreError: 348140:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [10509.923876] LustreError: 348140:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10509.928606] LustreError: 348140:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10509.933686] LustreError: 348140:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -22 [10525.720240] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10528.009643] Key type lgssc unregistered [10528.278435] LNet: 349126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10528.282987] LNetError: 349126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10528.298765] LNet: Removed LNI 192.168.203.134@tcp [10528.916486] Key type .llcrypt unregistered [10528.917731] Key type ._llcrypt unregistered [10544.830694] Key type ._llcrypt registered [10544.832252] Key type .llcrypt registered [10544.923320] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [10555.718953] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10556.654056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10556.674137] alg: No test for adler32 (adler32-zlib) [10557.775430] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [10558.009666] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10559.673875] Key type lgssc registered [10560.721687] Lustre: Echo OBD driver; http://www.lustre.org/ [10586.996634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10587.021780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.274850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10588.305968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10588.380774] Lustre: lustre-MDT0000: new disk, initializing [10588.431326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10588.442552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10591.678595] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10599.977741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.055177] Lustre: 352227:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10600.096635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10600.110225] Lustre: Skipped 1 previous similar message [10600.190414] Lustre: lustre-MDT0001: new disk, initializing [10600.272181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10600.301029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10600.305504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10603.390315] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10609.681288] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10612.626098] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10617.894843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10618.134704] Lustre: lustre-OST0000: new disk, initializing [10618.139558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10618.232576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10623.794469] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10627.651391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10627.669961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10627.764715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10629.562524] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10633.106246] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10633.298186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10637.041937] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10637.187930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10643.433370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10643.433520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.438057] LustreError: Skipped 1 previous similar message [10643.439923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10643.448965] Lustre: Skipped 1 previous similar message [10648.548646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10648.555081] Lustre: Skipped 2 previous similar messages [10653.154409] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10653.278690] Lustre: server umount lustre-OST0000 complete [10656.737354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10656.752809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10656.771134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10658.785938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10658.788343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.207329] Lustre: server umount lustre-MDT0000 complete [10665.209408] LustreError: 352218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617337 with bad export cookie 15627780277754983850 [10665.217855] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10665.220064] LustreError: 352218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10665.677032] Lustre: server umount lustre-MDT0001 complete [10672.138513] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10673.685578] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:29:05 (1773617345) [10679.292789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.605393] LustreError: 356280:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.619187] LustreError: 356280:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [10679.676095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.415643] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10684.915078] LustreError: 356281:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10690.026916] LustreError: 356280:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.351524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.700568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10696.196557] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10700.826359] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10703.953488] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10709.519508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10709.832290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.640452] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10717.718130] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10723.008031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10723.157289] Lustre: lustre-OST0001: new disk, initializing [10723.160081] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10725.599611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10725.609554] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10725.672269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10727.672738] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10733.273536] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10736.098366] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10736.099366] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10736.106717] LustreError: Skipped 1 previous similar message [10736.110097] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10736.117898] Lustre: Skipped 1 previous similar message [10736.121028] Lustre: Skipped 1 previous similar message [10742.221454] Lustre: server umount lustre-OST0001 complete [10742.252138] LustreError: 358961:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.272826] LustreError: 358961:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10747.375294] LustreError: 357875:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.388580] LustreError: 357875:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10757.601852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10757.611197] LustreError: 357875:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10757.612961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10757.618153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10757.618162] Lustre: Skipped 4 previous similar messages [10757.657402] LustreError: 357875:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [10763.473182] Lustre: server umount lustre-OST0000 complete [10767.847735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.863970] Lustre: Skipped 1 previous similar message [10772.588829] Lustre: server umount lustre-MDT0000 complete [10775.197480] LustreError: 356262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617447 with bad export cookie 15627780277754984739 [10775.201411] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10775.206797] LustreError: 356262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10781.536633] Lustre: server umount lustre-MDT0001 complete [10789.463587] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [10791.294262] Key type lgssc unregistered [10791.478496] LNet: 360636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10791.483605] LNetError: 360636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10791.507825] LNet: Removed LNI 192.168.203.134@tcp [10792.074902] Key type .llcrypt unregistered [10792.078675] Key type ._llcrypt unregistered [10828.313799] Key type ._llcrypt registered [10828.316407] Key type .llcrypt registered [10828.414113] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [10829.166358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10829.236539] alg: No test for adler32 (adler32-zlib) [10830.230850] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [10830.461730] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [10832.112508] Key type lgssc registered [10832.854732] Lustre: Echo OBD driver; http://www.lustre.org/ [10837.909777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10837.920844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.101214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10839.116925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10839.235754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10842.040440] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10847.190446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.259570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10847.288549] Lustre: 363395:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10847.431176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10850.095871] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10853.472045] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10855.646321] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10857.954248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10857.960234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10857.966353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10858.981856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10858.984820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.097566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.099593] Lustre: Skipped 2 previous similar messages [10869.218991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10869.238546] Lustre: Skipped 1 previous similar message [10871.264187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10871.481732] Lustre: server umount lustre-MDT0000 complete [10874.169407] LustreError: 363385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617546 with bad export cookie 13266542413832991730 [10874.177065] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10874.182071] LustreError: 363385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10874.428824] Lustre: server umount lustre-MDT0001 complete [10880.192218] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:32:32 (1773617552) [10885.237528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10885.610383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10888.495142] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10890.721696] LustreError: 365680:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.749138] LustreError: 365680:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10894.200462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.451550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10897.065034] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10900.270734] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10902.428215] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10906.662853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10906.789220] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10906.922549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10911.148564] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10914.785856] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10925.415703] Lustre: Permanently deactivating lustre-OST0000 [10925.426696] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10931.724325] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10933.430990] LustreError: 368323:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10933.441400] LustreError: 368323:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [10933.446709] LustreError: 368323:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10935.417762] LustreError: 368371:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [10935.424973] LustreError: 368371:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10943.194254] Lustre: server umount lustre-OST0000 complete [10945.036387] LustreError: 368621:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [10945.041662] LustreError: 368621:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10946.529765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.537152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10946.537894] Lustre: Skipped 1 previous similar message [10946.544207] Lustre: Skipped 1 previous similar message [10955.801626] Lustre: server umount lustre-MDT0000 complete [10956.769081] LustreError: 365680:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.782287] LustreError: 365680:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10958.327201] LustreError: 368025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617630 with bad export cookie 13266542413832992297 [10958.328720] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10958.336402] LustreError: 368025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10967.655242] Lustre: server umount lustre-MDT0001 complete [10972.896968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.327869] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10977.101336] LustreError: 369762:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10981.285467] Lustre: 370100:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.134@tcp [10982.682121] Lustre: server umount MGS complete [10989.995692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10990.254874] LustreError: 370821:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.266170] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10990.269445] Lustre: Skipped 1 previous similar message [10990.317848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10992.862918] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [10995.681395] LustreError: 370822:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.385787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.555435] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11001.003539] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11004.174954] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11006.558668] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11007.708976] Lustre: Permanently reactivating lustre-OST0000 [11007.717615] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11007.722022] Lustre: Skipped 2 previous similar messages [11028.448105] LustreError: 372198:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11032.845150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11033.016446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11033.019728] Lustre: Skipped 1 previous similar message [11036.335257] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11039.526505] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11048.416883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11048.422989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.435479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11048.440689] Lustre: Skipped 2 previous similar messages [11049.448736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11053.655826] Lustre: server umount lustre-OST0000 complete [11059.686835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.694088] Lustre: Skipped 2 previous similar messages [11062.011556] Lustre: server umount lustre-MDT0000 complete [11064.121313] LustreError: 370803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617736 with bad export cookie 13266542413832993347 [11064.123671] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11064.126693] LustreError: 370803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11064.296761] Lustre: server umount lustre-MDT0001 complete [11069.179423] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11070.606799] Key type lgssc unregistered [11070.778699] LNet: 374782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11070.783702] LNetError: 374782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11070.796864] LNet: Removed LNI 192.168.203.134@tcp [11071.261836] Key type .llcrypt unregistered [11071.264497] Key type ._llcrypt unregistered [11080.901445] Key type ._llcrypt registered [11080.903341] Key type .llcrypt registered [11080.964128] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [11088.923851] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11089.383431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11089.428550] alg: No test for adler32 (adler32-zlib) [11090.350126] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11090.476947] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11092.080232] Key type lgssc registered [11092.651957] Lustre: Echo OBD driver; http://www.lustre.org/ [11111.609311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11111.646244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.816791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11112.835578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11112.879392] Lustre: lustre-MDT0000: new disk, initializing [11112.921367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11112.939042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11114.920946] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11120.928387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.995675] Lustre: 377882:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11121.032078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11121.034884] Lustre: Skipped 1 previous similar message [11121.064792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11121.096156] Lustre: lustre-MDT0001: new disk, initializing [11121.182736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11121.214537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11121.221332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11123.911809] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11128.688755] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11130.838289] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11134.423106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11134.553128] Lustre: lustre-OST0000: new disk, initializing [11134.555509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11134.597177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11137.387446] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11141.249231] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11142.677670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11142.682754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11142.749271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11143.234804] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11143.333161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11145.411926] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11145.557495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11147.747855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11147.748689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11147.752720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11147.758178] Lustre: Skipped 1 previous similar message [11147.763207] Lustre: Skipped 1 previous similar message [11152.820449] Lustre: server umount lustre-OST0000 complete [11156.963542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11156.967133] LustreError: Skipped 1 previous similar message [11156.969872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.980954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11157.988551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11163.108243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11163.111067] Lustre: Skipped 3 previous similar messages [11168.228588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11168.231529] Lustre: Skipped 1 previous similar message [11169.248390] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11169.315330] LustreError: 381011:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000029ba66e4) refcount nonzero (1) after lock cleanup; forcing cleanup. [11171.384811] Lustre: server umount lustre-MDT0000 complete [11173.561876] LustreError: 377875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617846 with bad export cookie 17726879589741081190 [11173.565380] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11173.569226] LustreError: 377875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11173.760128] Lustre: server umount lustre-MDT0001 complete [11178.562617] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11179.555386] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:37:31 (1773617851) [11183.889765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.144591] LustreError: 381931:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.158666] LustreError: 381931:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11184.182666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.492633] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11189.218263] LustreError: 381932:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.310796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11191.592128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11193.759028] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11196.735539] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11198.704752] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11202.687046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.900456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11206.387371] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11209.208681] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11210.162917] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11212.927073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11222.500813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11222.501265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11222.521531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11226.725783] Lustre: server umount lustre-OST0000 complete [11232.737754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11232.738233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11232.755673] Lustre: Skipped 2 previous similar messages [11235.051875] Lustre: server umount lustre-MDT0000 complete [11236.838368] LustreError: 383545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617909 with bad export cookie 17726879589741082072 [11236.840820] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11236.842904] LustreError: 383545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11237.015772] Lustre: server umount lustre-MDT0001 complete [11241.595368] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11243.035965] Key type lgssc unregistered [11243.197589] LNet: 385299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11243.201147] LNetError: 385299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11243.213074] LNet: Removed LNI 192.168.203.134@tcp [11243.572143] Key type .llcrypt unregistered [11243.573570] Key type ._llcrypt unregistered [11255.079608] Key type ._llcrypt registered [11255.081599] Key type .llcrypt registered [11255.143864] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11255.282470] Key type .llcrypt unregistered [11255.285149] Key type ._llcrypt unregistered [11264.617312] Key type ._llcrypt registered [11264.618892] Key type .llcrypt registered [11264.676130] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11265.285983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11265.298231] alg: No test for adler32 (adler32-zlib) [11266.195914] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11266.317423] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11267.936491] Key type lgssc registered [11268.598748] Lustre: Echo OBD driver; http://www.lustre.org/ [11272.580366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11272.595079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11273.888128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11276.069434] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11280.998927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11281.204568] LustreError: 387453:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.212776] LustreError: 387453:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11281.215770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11284.326234] LustreError: 387452:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11284.340768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11284.343236] LustreError: 387452:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11284.387982] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11287.605652] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11289.573604] LustreError: 387453:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.587038] LustreError: 387453:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11291.241613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.430967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11293.397314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11304.929550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11304.932073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11304.937873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11306.980599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11306.980759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11306.986786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11308.900260] Lustre: server umount lustre-OST0000 complete [11312.097993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11312.098041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11312.104658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11312.104666] Lustre: Skipped 1 previous similar message [11312.109708] Lustre: Skipped 1 previous similar message [11316.939117] Lustre: server umount lustre-MDT0000 complete [11317.217459] LustreError: 387448:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11317.226124] LustreError: 387448:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11318.848781] LustreError: 389502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773617991 with bad export cookie 15116666422591943266 [11318.852076] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11318.856948] LustreError: 389502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11319.018346] Lustre: server umount lustre-MDT0001 complete [11323.766708] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11325.249927] Key type lgssc unregistered [11325.414545] LNet: 390436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11325.417507] LNetError: 390436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11325.425930] LNet: Removed LNI 192.168.203.134@tcp [11325.816176] Key type .llcrypt unregistered [11325.817851] Key type ._llcrypt unregistered [11339.086250] Key type ._llcrypt registered [11339.087807] Key type .llcrypt registered [11339.143354] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11339.531187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11339.540621] alg: No test for adler32 (adler32-zlib) [11340.435639] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11340.553036] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11342.144165] Key type lgssc registered [11342.676233] Lustre: Echo OBD driver; http://www.lustre.org/ [11346.751759] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11346.762876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11362.275333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11367.392226] LustreError: 391802:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [11394.016262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11396.948218] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11399.771259] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11403.541836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.830335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.868591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11406.037000] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11408.870039] LustreError: 391828:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.884277] LustreError: 391828:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11409.892517] LustreError: 392729:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.902379] LustreError: 392729:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11411.313984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.557794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11413.588133] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11416.452894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11418.452594] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11424.226504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11424.231393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11424.240103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11426.786113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11426.797957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11426.807247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11429.480372] Lustre: server umount lustre-OST0000 complete [11431.913500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11431.922702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11431.934085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11431.937881] Lustre: Skipped 2 previous similar messages [11433.771249] Lustre: server umount lustre-MDT0000 complete [11435.509405] LustreError: 394085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618108 with bad export cookie 11010476799209376406 [11435.513697] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11435.513756] LustreError: 394085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11435.660847] Lustre: server umount lustre-MDT0001 complete [11440.778870] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11442.084305] Key type lgssc unregistered [11442.237602] LNet: 395021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11442.241189] LNetError: 395021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11442.248486] LNet: Removed LNI 192.168.203.134@tcp [11442.656199] Key type .llcrypt unregistered [11442.657818] Key type ._llcrypt unregistered [11456.357871] Key type ._llcrypt registered [11456.359093] Key type .llcrypt registered [11456.437423] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11457.141806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11457.158972] alg: No test for adler32 (adler32-zlib) [11458.081625] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11458.187526] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11459.784235] Key type lgssc registered [11460.410969] Lustre: Echo OBD driver; http://www.lustre.org/ [11464.473522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11464.485083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.697452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11467.313412] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11471.510404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.779946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11473.761474] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11476.556346] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11478.396710] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11481.780468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.062794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11485.347696] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11487.644400] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11491.309572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11496.418719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.419589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11496.425904] Lustre: Skipped 2 previous similar messages [11496.430542] Lustre: Skipped 2 previous similar messages [11499.333467] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.34@tcp (stopping) [11501.241612] Lustre: server umount lustre-MDT0000 complete [11501.541495] LustreError: 396399:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.548415] LustreError: 396399:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [11502.220876] LustreError: 398638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.34@tcp arrived at 1773618174 with bad export cookie 24389222087900772 [11502.226167] LustreError: 398638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11503.067572] LustreError: 396378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618175 with bad export cookie 24389222087900142 [11503.071174] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11503.071777] LustreError: 396378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11506.657030] LustreError: 396398:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.657596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.667674] LustreError: 396398:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11506.675972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.708876] LustreError: 397506:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11508.709185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.716571] LustreError: 397506:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11509.216654] Lustre: server umount lustre-OST0000 complete [11511.932184] Lustre: server umount lustre-MDT0001 complete [11516.196546] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11517.375587] Key type lgssc unregistered [11517.526731] LNet: 399725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11517.530631] LNetError: 399725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11517.543989] LNet: Removed LNI 192.168.203.134@tcp [11517.915985] Key type .llcrypt unregistered [11517.917615] Key type ._llcrypt unregistered [11529.006727] Key type ._llcrypt registered [11529.008052] Key type .llcrypt registered [11529.053706] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11529.452337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11529.510440] alg: No test for adler32 (adler32-zlib) [11530.382858] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11530.469518] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11532.065796] Key type lgssc registered [11532.563167] Lustre: Echo OBD driver; http://www.lustre.org/ [11535.913738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11535.924215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11537.178448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.787969] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11542.930776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.151811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11544.967805] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11547.677964] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11549.300404] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11552.849238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11553.027497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11556.157353] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11558.642284] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11562.283188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11567.586925] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11567.593254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11567.595775] Lustre: Skipped 1 previous similar message [11570.504160] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [11572.658470] Lustre: server umount lustre-MDT0001 complete [11572.705770] LustreError: 401115:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11572.713185] LustreError: 401115:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11577.826085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.829156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11577.830381] LustreError: 401115:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.835854] Lustre: Skipped 1 previous similar message [11577.851277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11581.222772] Lustre: server umount lustre-OST0000 complete [11582.945202] LustreError: 401116:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.432630] Lustre: server umount lustre-MDT0000 complete [11590.061874] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11591.513361] Key type lgssc unregistered [11591.679041] LNet: 404440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11591.685872] LNetError: 404440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11591.695626] LNet: Removed LNI 192.168.203.134@tcp [11592.128306] Key type .llcrypt unregistered [11592.129763] Key type ._llcrypt unregistered [11604.096231] Key type ._llcrypt registered [11604.098297] Key type .llcrypt registered [11604.155027] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [11612.055195] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11612.584255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11612.615837] alg: No test for adler32 (adler32-zlib) [11613.545706] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11613.683040] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11615.296213] Key type lgssc registered [11615.939415] Lustre: Echo OBD driver; http://www.lustre.org/ [11634.370511] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11643.237235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11643.250851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.432325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11644.457492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11644.506531] Lustre: lustre-MDT0000: new disk, initializing [11644.566155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11644.579691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11646.525820] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11655.313793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.490433] Lustre: lustre-OST0000: new disk, initializing [11655.493728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11655.499058] Lustre: Skipped 1 previous similar message [11655.540575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11658.851706] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11660.837440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11660.843454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11660.860335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11663.390272] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11671.848402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.920196] Lustre: 408378:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11671.964056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11672.004819] Lustre: lustre-MDT0001: new disk, initializing [11672.070465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11672.086714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11672.090951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11674.164838] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11679.800422] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11679.946685] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11682.286863] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11682.398278] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11686.217635] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [11686.882900] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11686.887214] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.896401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11686.899468] Lustre: Skipped 1 previous similar message [11691.158188] Lustre: server umount lustre-MDT0001 complete [11700.194613] LustreError: 408413:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11700.204178] LustreError: 408413:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [11705.316037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11705.316274] LustreError: 410602:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.334474] Lustre: Skipped 1 previous similar message [11705.351587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11710.435058] LustreError: 408389:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11710.435767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.556672] LustreError: 410602:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11715.559410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11719.137378] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11719.227629] Lustre: server umount lustre-OST0000 complete [11720.680949] LustreError: 408389:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.653230] Lustre: server umount lustre-MDT0000 complete [11733.833476] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11735.278336] Key type lgssc unregistered [11735.446607] LNet: 412942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11735.451975] LNetError: 412942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11735.461369] LNet: Removed LNI 192.168.203.134@tcp [11735.894720] Key type .llcrypt unregistered [11735.896474] Key type ._llcrypt unregistered [11755.120883] Key type ._llcrypt registered [11755.122408] Key type .llcrypt registered [11755.168351] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [11755.298472] Key type .llcrypt unregistered [11755.299871] Key type ._llcrypt unregistered [11763.655607] Key type ._llcrypt registered [11763.656950] Key type .llcrypt registered [11763.719549] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [11764.208567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11764.230517] alg: No test for adler32 (adler32-zlib) [11765.104316] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [11765.207452] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [11766.808156] Key type lgssc registered [11767.318640] Lustre: Echo OBD driver; http://www.lustre.org/ [11778.987628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11779.001955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11780.115556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11780.132304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11780.173201] Lustre: lustre-MDT0000: new disk, initializing [11780.214433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11780.222185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11781.830879] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11786.770407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11786.819183] Lustre: 416259:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11786.857844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11786.860015] Lustre: Skipped 1 previous similar message [11786.898205] Lustre: lustre-MDT0001: new disk, initializing [11786.935769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11786.950458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11786.954824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11788.523588] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11792.678725] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11794.406210] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11797.301926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11797.453502] Lustre: lustre-OST0000: new disk, initializing [11797.456083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11797.507145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11799.838796] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11801.605766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11801.612247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11801.661079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11803.701501] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11807.714111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11807.717389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11807.724580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11811.810238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11811.811034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11811.819530] Lustre: Skipped 1 previous similar message [11811.824751] Lustre: Skipped 2 previous similar messages [11813.346950] Lustre: server umount lustre-MDT0000 complete [11814.898092] LustreError: 417037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618487 with bad export cookie 10010005092455607354 [11814.900163] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11814.907046] LustreError: 417037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11815.034398] Lustre: server umount lustre-MDT0001 complete [11827.048486] Lustre: server umount lustre-OST0000 complete [11831.446855] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:48:23 (1773618503) [11835.019164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.219799] LustreError: 419910:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.234337] LustreError: 419910:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11835.251586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.923310] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11840.481229] LustreError: 419911:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11840.494730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11840.667982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11842.194571] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11844.458649] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11845.913883] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11862.052518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11862.132971] Lustre: Found index 0 for lustre-OST0000, updating log [11862.210058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11864.511350] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [11866.564901] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11879.655427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11885.026658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11885.027868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.036974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11885.849357] Lustre: server umount lustre-OST0000 complete [11900.963121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.967911] Lustre: Skipped 1 previous similar message [11900.970559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11900.972557] Lustre: Skipped 2 previous similar messages [11907.041065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11907.044754] Lustre: Skipped 3 previous similar messages [11913.184161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11913.246133] Lustre: server umount lustre-MDT0000 complete [11914.786243] LustreError: 419891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773618587 with bad export cookie 10010005092455608747 [11914.789644] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11914.791740] LustreError: 419891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11914.930335] Lustre: server umount lustre-MDT0001 complete [11918.710555] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:49:50 (1773618590) [11927.010034] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [12503.423401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12503.602163] LustreError: 426000:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.634186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12505.081147] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12508.466974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.644695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12510.208784] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12512.440398] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12513.749513] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12529.639208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12529.752486] Lustre: Found index 0 for lustre-OST0000, updating log [12529.869758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12531.947496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12532.166863] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12534.618960] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12536.481974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.482754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12536.486634] Lustre: Skipped 1 previous similar message [12536.488819] Lustre: Skipped 2 previous similar messages [12541.596254] Lustre: server umount lustre-OST0000 complete [12543.229350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.230104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12543.234309] Lustre: Skipped 1 previous similar message [12543.239051] Lustre: Skipped 3 previous similar messages [12545.315199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12545.317743] Lustre: Skipped 2 previous similar messages [12549.409305] LustreError: 426019:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.416478] LustreError: 426019:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12549.422350] Lustre: server umount lustre-MDT0000 complete [12550.435282] LustreError: 426001:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12550.442709] LustreError: 426001:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12551.196320] LustreError: 425982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619223 with bad export cookie 10010005092455609818 [12551.199229] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12551.201577] LustreError: 425982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12551.326955] Lustre: server umount lustre-MDT0001 complete [12554.720410] LNet: 1 local NIs in recovery (showing 1): 192.168.203.134@tcp [12570.386827] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [12573.984906] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12589.279258] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:01:01 (1773619261) [12596.130090] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12597.243594] Key type lgssc unregistered [12597.373608] LNet: 432249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12597.376985] LNetError: 432249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12661.926319] LNet: Removed LNI 192.168.203.134@tcp [12662.276676] Key type .llcrypt unregistered [12662.278047] Key type ._llcrypt unregistered [12669.461181] Key type ._llcrypt registered [12669.462666] Key type .llcrypt registered [12669.503830] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12669.846925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12669.882577] alg: No test for adler32 (adler32-zlib) [12670.744139] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [12670.833556] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12672.416148] Key type lgssc registered [12672.835307] Lustre: Echo OBD driver; http://www.lustre.org/ [12674.380946] LNet: Added LNI 192.168.203.134@tcp100 [8/256/0/180] [12674.698625] LNet: Added LNI 192.168.203.134@tcp101 [8/256/0/180] [12675.021206] LNet: Added LNI 192.168.203.134@tcp102 [8/256/0/180] [12675.327350] LNet: Added LNI 192.168.203.134@tcp103 [8/256/0/180] [12675.634980] LNet: Added LNI 192.168.203.134@tcp104 [8/256/0/180] [12675.937083] LNet: Added LNI 192.168.203.134@tcp105 [8/256/0/180] [12676.224629] LNet: Added LNI 192.168.203.134@tcp106 [8/256/0/180] [12676.514646] LNet: Added LNI 192.168.203.134@tcp107 [8/256/0/180] [12676.810880] LNet: Added LNI 192.168.203.134@tcp108 [8/256/0/180] [12677.101956] LNet: Added LNI 192.168.203.134@tcp109 [8/256/0/180] [12677.394312] LNet: Added LNI 192.168.203.134@tcp110 [8/256/0/180] [12677.687844] LNet: Added LNI 192.168.203.134@tcp111 [8/256/0/180] [12678.008746] LNet: Added LNI 192.168.203.134@tcp112 [8/256/0/180] [12678.342582] LNet: Added LNI 192.168.203.134@tcp113 [8/256/0/180] [12678.696443] LNet: Added LNI 192.168.203.134@tcp114 [8/256/0/180] [12679.034389] LNet: Added LNI 192.168.203.134@tcp115 [8/256/0/180] [12679.375390] LNet: Added LNI 192.168.203.134@tcp116 [8/256/0/180] [12679.711497] LNet: Added LNI 192.168.203.134@tcp117 [8/256/0/180] [12680.089841] LNet: Added LNI 192.168.203.134@tcp118 [8/256/0/180] [12680.466416] LNet: Added LNI 192.168.203.134@tcp119 [8/256/0/180] [12680.819417] LNet: Added LNI 192.168.203.134@tcp120 [8/256/0/180] [12681.226418] LNet: Added LNI 192.168.203.134@tcp121 [8/256/0/180] [12681.609718] LNet: Added LNI 192.168.203.134@tcp122 [8/256/0/180] [12682.017738] LNet: Added LNI 192.168.203.134@tcp123 [8/256/0/180] [12682.364657] LNet: Added LNI 192.168.203.134@tcp124 [8/256/0/180] [12682.697146] LNet: Added LNI 192.168.203.134@tcp125 [8/256/0/180] [12683.031549] LNet: Added LNI 192.168.203.134@tcp126 [8/256/0/180] [12683.341616] LNet: Added LNI 192.168.203.134@tcp127 [8/256/0/180] [12683.704129] LNet: Added LNI 192.168.203.134@tcp128 [8/256/0/180] [12684.086914] LNet: Added LNI 192.168.203.134@tcp129 [8/256/0/180] [12684.486527] LNet: Added LNI 192.168.203.134@tcp130 [8/256/0/180] [12684.858731] LNet: Added LNI 192.168.203.134@tcp131 [8/256/0/180] [12685.294381] LNet: Added LNI 192.168.203.134@tcp132 [8/256/0/180] [12685.726792] LNet: Added LNI 192.168.203.134@tcp133 [8/256/0/180] [12686.097561] LNet: Added LNI 192.168.203.134@tcp134 [8/256/0/180] [12686.443577] LNet: Added LNI 192.168.203.134@tcp135 [8/256/0/180] [12688.916671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12688.923403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.972235] Lustre: 435228:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [12689.993625] LNetError: 432694:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [12690.000088] LNetError: Protocol error connecting to 192.168.203.134@tcp114 at host 192.168.203.134:988: is it running a compatible version of Lustre? [12690.055116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12690.068445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12690.109722] Lustre: lustre-MDT0000: new disk, initializing [12690.170639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12690.178641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12691.616795] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12696.620951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.679213] Lustre: 435246:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12696.706792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12696.709221] Lustre: Skipped 1 previous similar message [12696.746750] Lustre: lustre-MDT0001: new disk, initializing [12696.761797] Lustre: 436031:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [12696.765132] Lustre: 436031:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12696.808464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12696.821365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12696.824954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12698.187791] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12698.551399] Lustre: 436330:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [12698.555972] Lustre: 436330:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12701.535179] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12702.744094] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12705.252514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12705.371581] Lustre: lustre-OST0000: new disk, initializing [12705.373763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12705.408722] Lustre: 436997:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.134@tcp: can't add NID 192.168.203.134@tcp130: rc = -75 [12705.412654] Lustre: 436997:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [12705.431427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12707.505511] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12708.424516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12708.428356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12708.470436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12710.807142] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12713.954473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12713.954516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12713.957121] LustreError: Skipped 1 previous similar message [12713.962583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12713.966265] Lustre: Skipped 1 previous similar message [12719.075769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12719.077893] Lustre: Skipped 1 previous similar message [12719.774352] Lustre: server umount lustre-OST0000 complete [12720.993357] LNetError: 432694:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [12720.997730] LNetError: 432694:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 2 previous similar messages [12721.001569] LNetError: Protocol error connecting to 192.168.203.134@tcp114 at host 192.168.203.134:988: is it running a compatible version of Lustre? [12721.006277] LNetError: Skipped 2 previous similar messages [12722.656823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12722.660092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.664889] Lustre: Skipped 1 previous similar message [12722.667056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12723.041454] LNetError: 432695:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [12723.044919] LNetError: Protocol error connecting to 192.168.203.134@tcp112 at host 192.168.203.134:988: is it running a compatible version of Lustre? [12724.193869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12727.451574] Lustre: server umount lustre-MDT0000 complete [12728.990392] LustreError: 435236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619401 with bad export cookie 8199387913679859303 [12728.992868] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12728.997304] LustreError: 435236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12729.128716] Lustre: server umount lustre-MDT0001 complete [12731.233158] LNetError: 435252:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.134 [12731.236902] LNetError: Protocol error connecting to 192.168.203.134@tcp111 at host 192.168.203.134:988: is it running a compatible version of Lustre? [12733.067690] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12734.144304] Key type lgssc unregistered [12734.310543] LNet: 438775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12734.313765] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12734.317176] LNet: Removed LNI 192.168.203.134@tcp [12735.328125] LNet: Removed LNI 192.168.203.134@tcp100 [12735.329716] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12735.332701] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12735.335711] LNet: Removed LNI 192.168.203.134@tcp101 [12736.352160] LNet: Removed LNI 192.168.203.134@tcp102 [12736.353937] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12736.356994] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12736.360538] LNet: Removed LNI 192.168.203.134@tcp103 [12737.376141] LNet: Removed LNI 192.168.203.134@tcp104 [12738.400191] LNet: Removed LNI 192.168.203.134@tcp105 [12738.402475] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12738.406278] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12738.409606] LNet: Removed LNI 192.168.203.134@tcp106 [12739.424167] LNet: Removed LNI 192.168.203.134@tcp107 [12740.448141] LNet: Removed LNI 192.168.203.134@tcp108 [12741.472179] LNet: Removed LNI 192.168.203.134@tcp109 [12742.496130] LNet: Removed LNI 192.168.203.134@tcp110 [12742.497863] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12742.500395] LNetError: 438775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12742.503225] LNet: Removed LNI 192.168.203.134@tcp111 [12743.520164] LNet: Removed LNI 192.168.203.134@tcp112 [12744.544142] LNet: Removed LNI 192.168.203.134@tcp113 [12745.568192] LNet: Removed LNI 192.168.203.134@tcp114 [12746.592136] LNet: Removed LNI 192.168.203.134@tcp115 [12747.616168] LNet: Removed LNI 192.168.203.134@tcp116 [12748.640140] LNet: Removed LNI 192.168.203.134@tcp117 [12749.664149] LNet: Removed LNI 192.168.203.134@tcp118 [12749.665797] LNet: Removed LNI 192.168.203.134@tcp119 [12749.667201] LNet: Removed LNI 192.168.203.134@tcp120 [12749.668694] LNet: Removed LNI 192.168.203.134@tcp121 [12749.669992] LNet: Removed LNI 192.168.203.134@tcp122 [12749.671266] LNet: Removed LNI 192.168.203.134@tcp123 [12749.672521] LNet: Removed LNI 192.168.203.134@tcp124 [12749.673866] LNet: Removed LNI 192.168.203.134@tcp125 [12749.675428] LNet: Removed LNI 192.168.203.134@tcp126 [12749.677033] LNet: Removed LNI 192.168.203.134@tcp127 [12749.678869] LNet: Removed LNI 192.168.203.134@tcp128 [12749.680640] LNet: Removed LNI 192.168.203.134@tcp129 [12749.682985] LNet: Removed LNI 192.168.203.134@tcp130 [12749.684605] LNet: Removed LNI 192.168.203.134@tcp131 [12749.686053] LNet: Removed LNI 192.168.203.134@tcp132 [12749.687619] LNet: Removed LNI 192.168.203.134@tcp133 [12749.689222] LNet: Removed LNI 192.168.203.134@tcp134 [12749.696496] LNet: Removed LNI 192.168.203.134@tcp135 [12750.010944] Key type .llcrypt unregistered [12750.013347] Key type ._llcrypt unregistered [12759.065523] Key type ._llcrypt registered [12759.067061] Key type .llcrypt registered [12759.103683] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [12765.504844] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12765.858850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12765.891411] alg: No test for adler32 (adler32-zlib) [12766.748314] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [12766.838626] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12768.424147] Key type lgssc registered [12768.843914] Lustre: Echo OBD driver; http://www.lustre.org/ [12781.357611] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:04:13 (1773619453) [12787.759708] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12788.808769] Key type lgssc unregistered [12788.944668] LNet: 442781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12788.947707] LNetError: 442781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12788.955404] LNet: Removed LNI 192.168.203.134@tcp [12789.246437] Key type .llcrypt unregistered [12789.247963] Key type ._llcrypt unregistered [12796.164571] Key type ._llcrypt registered [12796.165799] Key type .llcrypt registered [12796.202616] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12796.538966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12796.594517] alg: No test for adler32 (adler32-zlib) [12797.444174] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [12797.526533] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12799.112158] Key type lgssc registered [12799.476195] Lustre: Echo OBD driver; http://www.lustre.org/ [12802.713222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12802.720972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12803.813558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12803.826121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12803.865352] Lustre: lustre-MDT0000: new disk, initializing [12803.891632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12803.900968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12805.232927] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12810.033994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12810.087936] Lustre: 443953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12810.118196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12810.120992] Lustre: Skipped 1 previous similar message [12810.163390] Lustre: lustre-MDT0001: new disk, initializing [12810.206154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12810.226074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12810.229579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12811.775371] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12814.997794] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12816.305894] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12818.831048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12818.936461] Lustre: lustre-OST0000: new disk, initializing [12818.938652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12818.969113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12820.285699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12820.293395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12820.325456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12821.075545] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12824.577614] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12825.772958] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing lnet_if_list [12827.668747] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12827.681674] LNet: Added LNI 192.168.203.134@tcp10 [8/256/0/180] [12829.565308] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12829.578565] LNet: Added LNI 192.168.203.134@tcp11 [8/256/0/180] [12830.934689] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12830.946412] LNet: Added LNI 192.168.203.134@tcp12 [8/256/0/180] [12832.276057] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12832.286867] LNet: Added LNI 192.168.203.134@tcp13 [8/256/0/180] [12833.607989] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12833.619629] LNet: Added LNI 192.168.203.134@tcp14 [8/256/0/180] [12835.023369] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12835.035421] LNet: Added LNI 192.168.203.134@tcp15 [8/256/0/180] [12836.497772] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12836.510555] LNet: Added LNI 192.168.203.134@tcp16 [8/256/0/180] [12838.182452] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12838.196739] LNet: Added LNI 192.168.203.134@tcp17 [8/256/0/180] [12839.682135] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12839.694027] LNet: Added LNI 192.168.203.134@tcp18 [8/256/0/180] [12841.215622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12841.226499] LNet: Added LNI 192.168.203.134@tcp19 [8/256/0/180] [12842.802824] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12842.815365] LNet: Added LNI 192.168.203.134@tcp20 [8/256/0/180] [12844.396633] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12844.410940] LNet: Added LNI 192.168.203.134@tcp21 [8/256/0/180] [12846.068200] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12846.079359] LNet: Added LNI 192.168.203.134@tcp22 [8/256/0/180] [12847.710218] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12847.722624] LNet: Added LNI 192.168.203.134@tcp23 [8/256/0/180] [12849.413563] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12849.425532] LNet: Added LNI 192.168.203.134@tcp24 [8/256/0/180] [12851.141939] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12851.153365] LNet: Added LNI 192.168.203.134@tcp25 [8/256/0/180] [12852.955794] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12852.967228] LNet: Added LNI 192.168.203.134@tcp26 [8/256/0/180] [12854.758502] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12854.769932] LNet: Added LNI 192.168.203.134@tcp27 [8/256/0/180] [12856.635696] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12856.646518] LNet: Added LNI 192.168.203.134@tcp28 [8/256/0/180] [12858.500881] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12858.513421] LNet: Added LNI 192.168.203.134@tcp29 [8/256/0/180] [12860.457190] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12860.468616] LNet: Added LNI 192.168.203.134@tcp30 [8/256/0/180] [12862.539227] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12862.553926] LNet: Added LNI 192.168.203.134@tcp31 [8/256/0/180] [12864.831808] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12864.844527] LNet: Added LNI 192.168.203.134@tcp32 [8/256/0/180] [12867.003048] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12867.016800] LNet: Added LNI 192.168.203.134@tcp33 [8/256/0/180] [12869.217877] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12869.231494] LNet: Added LNI 192.168.203.134@tcp34 [8/256/0/180] [12871.424579] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12871.436878] LNet: Added LNI 192.168.203.134@tcp35 [8/256/0/180] [12873.611135] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12873.623330] LNet: Added LNI 192.168.203.134@tcp36 [8/256/0/180] [12875.827632] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12875.840867] LNet: Added LNI 192.168.203.134@tcp37 [8/256/0/180] [12878.038085] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12878.049959] LNet: Added LNI 192.168.203.134@tcp38 [8/256/0/180] [12880.366448] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12880.380086] LNet: Added LNI 192.168.203.134@tcp39 [8/256/0/180] [12882.886570] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12882.898714] LNet: Added LNI 192.168.203.134@tcp40 [8/256/0/180] [12885.213935] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12885.225283] LNet: Added LNI 192.168.203.134@tcp41 [8/256/0/180] [12887.608421] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12887.621710] LNet: Added LNI 192.168.203.134@tcp42 [8/256/0/180] [12890.148640] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12890.167627] LNet: Added LNI 192.168.203.134@tcp43 [8/256/0/180] [12892.730466] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12892.747410] LNet: Added LNI 192.168.203.134@tcp44 [8/256/0/180] [12895.489436] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12895.502757] LNet: Added LNI 192.168.203.134@tcp45 [8/256/0/180] [12898.294150] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12898.307206] LNet: Added LNI 192.168.203.134@tcp46 [8/256/0/180] [12901.163716] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12901.175197] LNet: Added LNI 192.168.203.134@tcp47 [8/256/0/180] [12903.909548] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12903.922154] LNet: Added LNI 192.168.203.134@tcp48 [8/256/0/180] [12906.789361] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12906.802730] LNet: Added LNI 192.168.203.134@tcp49 [8/256/0/180] [12909.802758] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12909.814660] LNet: Added LNI 192.168.203.134@tcp50 [8/256/0/180] [12917.730454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12917.730535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.731792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12917.739572] Lustre: Skipped 2 previous similar messages [12922.849505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.852116] Lustre: Skipped 4 previous similar messages [12923.614470] Lustre: server umount lustre-MDT0000 complete [12924.972134] LustreError: 445697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619597 with bad export cookie 757194410578099283 [12924.975332] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12924.977581] LustreError: 445697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12925.092559] Lustre: server umount lustre-MDT0001 complete [12932.478774] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12935.542177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12944.352154] Lustre: 443230:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773619600/real 1773619600] req@0000000052f5142a x1859774809457536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773619616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12944.360401] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12946.400114] Lustre: 443227:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773619602/real 1773619602] req@0000000078129678 x1859774809457664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773619618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12948.448128] Lustre: 443228:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773619604/real 1773619604] req@00000000d2525017 x1859774809457792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773619620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12949.472421] LustreError: 443226:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000695a3268 x1859774809458560/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12949.596299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12950.496094] Lustre: 443229:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773619607/real 1773619607] req@000000001bfdef48 x1859774809458048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773619623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12950.961344] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12953.861665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12953.991017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12955.046146] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12955.362265] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [12956.064106] Lustre: 443230:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773619612/real 1773619612] req@00000000ce9aa324 x1859774809458176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773619628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12957.209943] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12958.428888] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12969.441133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12969.443470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12969.447564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12972.120291] Lustre: server umount lustre-OST0000 complete [12974.560870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12974.563968] LustreError: Skipped 1 previous similar message [12974.566064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12974.574368] Lustre: Skipped 1 previous similar message [12974.576215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12974.578452] Lustre: Skipped 3 previous similar messages [12979.608978] Lustre: server umount lustre-MDT0000 complete [12980.705627] LustreError: 470881:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12980.711610] LustreError: 470881:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [12981.125381] LustreError: 443944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619653 with bad export cookie 757194410578101369 [12981.128083] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12981.129417] LustreError: 443944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12981.251105] Lustre: server umount lustre-MDT0001 complete [12985.109125] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [12986.331318] Key type lgssc unregistered [12986.504980] LNet: 471622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12986.508811] LNetError: 471622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12986.511512] LNet: Removed LNI 192.168.203.134@tcp [12986.512969] LNet: Removed LNI 192.168.203.134@tcp10 [12986.514265] LNet: Removed LNI 192.168.203.134@tcp11 [12986.515753] LNet: Removed LNI 192.168.203.134@tcp12 [12986.517240] LNet: Removed LNI 192.168.203.134@tcp13 [12986.518890] LNet: Removed LNI 192.168.203.134@tcp14 [12986.520417] LNet: Removed LNI 192.168.203.134@tcp15 [12986.521957] LNet: Removed LNI 192.168.203.134@tcp16 [12986.523449] LNet: Removed LNI 192.168.203.134@tcp17 [12986.525780] LNet: Removed LNI 192.168.203.134@tcp18 [12986.527702] LNet: Removed LNI 192.168.203.134@tcp19 [12986.529577] LNet: Removed LNI 192.168.203.134@tcp20 [12986.531088] LNet: Removed LNI 192.168.203.134@tcp21 [12986.532911] LNet: Removed LNI 192.168.203.134@tcp22 [12986.534091] LNet: Removed LNI 192.168.203.134@tcp23 [12986.535636] LNet: Removed LNI 192.168.203.134@tcp24 [12986.537309] LNet: Removed LNI 192.168.203.134@tcp25 [12986.538985] LNet: Removed LNI 192.168.203.134@tcp26 [12986.540754] LNet: Removed LNI 192.168.203.134@tcp27 [12986.542492] LNet: Removed LNI 192.168.203.134@tcp28 [12986.544504] LNet: Removed LNI 192.168.203.134@tcp29 [12986.546161] LNet: Removed LNI 192.168.203.134@tcp30 [12986.547905] LNet: Removed LNI 192.168.203.134@tcp31 [12986.549627] LNet: Removed LNI 192.168.203.134@tcp32 [12986.551236] LNet: Removed LNI 192.168.203.134@tcp33 [12986.552965] LNet: Removed LNI 192.168.203.134@tcp34 [12986.554504] LNet: Removed LNI 192.168.203.134@tcp35 [12986.555983] LNet: Removed LNI 192.168.203.134@tcp36 [12986.557600] LNet: Removed LNI 192.168.203.134@tcp37 [12986.559273] LNet: Removed LNI 192.168.203.134@tcp38 [12986.561349] LNet: Removed LNI 192.168.203.134@tcp39 [12986.563351] LNet: Removed LNI 192.168.203.134@tcp40 [12986.564803] LNet: Removed LNI 192.168.203.134@tcp41 [12986.566239] LNet: Removed LNI 192.168.203.134@tcp42 [12986.567972] LNet: Removed LNI 192.168.203.134@tcp43 [12986.569181] LNet: Removed LNI 192.168.203.134@tcp44 [12986.570470] LNet: Removed LNI 192.168.203.134@tcp45 [12986.571993] LNet: Removed LNI 192.168.203.134@tcp46 [12986.573435] LNet: Removed LNI 192.168.203.134@tcp47 [12986.574734] LNet: Removed LNI 192.168.203.134@tcp48 [12986.576188] LNet: Removed LNI 192.168.203.134@tcp49 [12986.583344] LNet: Removed LNI 192.168.203.134@tcp50 [12986.927487] Key type .llcrypt unregistered [12986.929048] Key type ._llcrypt unregistered [12995.361828] Key type ._llcrypt registered [12995.363288] Key type .llcrypt registered [12995.408951] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [12995.872749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12995.879933] alg: No test for adler32 (adler32-zlib) [12996.736227] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [12996.825951] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [12998.408264] Key type lgssc registered [12998.817734] Lustre: Echo OBD driver; http://www.lustre.org/ [13001.678583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13001.685546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13002.854740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13004.378865] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13007.775369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13007.969793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13009.531647] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13011.694739] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13013.102637] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13015.668046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13015.839220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13017.929453] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13019.893987] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13028.138184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13034.089116] loop0: detected capacity change from 0 to 204800000 [13034.145801] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13034.150565] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13034.155795] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13036.421369] loop0: detected capacity change from 0 to 204800000 [13036.501535] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13036.506457] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13036.511352] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13038.533204] loop0: detected capacity change from 0 to 204800000 [13040.728814] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13040.789769] Lustre: fs15246-OST0000: new disk, initializing [13040.792475] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13040.819971] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13043.108462] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13047.373356] loop1: detected capacity change from 0 to 204800000 [13049.378861] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13049.437416] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13049.466156] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13049.509462] Lustre: fs15246-MDT0000: new disk, initializing [13049.578906] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13049.592648] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13049.632536] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13050.656960] Lustre: 477225:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000dd3b38cf x1859775018073984/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773619739 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13051.159453] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13059.634719] LustreError: 477225:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [13059.640774] LustreError: 477225:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [13059.644585] LustreError: 477225:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [13061.093596] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.099731] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13065.763138] Lustre: server umount fs15246-MDT0000 complete [13066.214546] LustreError: 477226:0:(ldlm_lib.c:1178:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.220837] LustreError: 477226:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13069.344490] Lustre: server umount fs15246-OST0000 complete [13074.401353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13074.401561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13074.403279] LustreError: Skipped 1 previous similar message [13074.404757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13074.408185] Lustre: Skipped 1 previous similar message [13074.414480] Lustre: Skipped 1 previous similar message [13076.449438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13079.322176] Lustre: server umount lustre-OST0000 complete [13081.570137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.571508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13081.575102] Lustre: Skipped 1 previous similar message [13081.578095] Lustre: Skipped 1 previous similar message [13086.689202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13086.691289] Lustre: Skipped 1 previous similar message [13095.392153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13095.528899] Lustre: server umount lustre-MDT0000 complete [13096.928981] LustreError: 473555:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13096.936888] LustreError: 473555:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13097.027983] LustreError: 474460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619769 with bad export cookie 12077637196836793284 [13097.031374] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13097.032728] LustreError: 474460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13097.152456] Lustre: server umount lustre-MDT0001 complete [13101.037258] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [13102.128657] Key type lgssc unregistered [13102.260484] LNet: 480288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13102.264159] LNetError: 480288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13102.272460] LNet: Removed LNI 192.168.203.134@tcp [13102.589161] Key type .llcrypt unregistered [13102.590515] Key type ._llcrypt unregistered [13110.383737] Key type ._llcrypt registered [13110.385120] Key type .llcrypt registered [13110.428365] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [13110.763222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13110.803064] alg: No test for adler32 (adler32-zlib) [13111.660740] Lustre: Lustre: Build Version: 2.17.51_2_gc98ff3f [13111.747346] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [13113.328194] Key type lgssc registered [13113.757379] Lustre: Echo OBD driver; http://www.lustre.org/ [13128.206987] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [13132.003681] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [13144.933653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13144.940202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.037602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13146.051866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13146.088498] Lustre: lustre-MDT0000: new disk, initializing [13146.113865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13146.121323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13147.430798] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13151.798307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13151.849217] Lustre: 484672:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13151.877486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13151.880185] Lustre: Skipped 1 previous similar message [13151.923225] Lustre: lustre-MDT0001: new disk, initializing [13151.968057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13151.987077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13151.990294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13153.331119] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13156.573834] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13157.792754] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13160.318382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13160.459244] Lustre: lustre-OST0000: new disk, initializing [13160.461750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13160.496540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13162.390314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [13163.899533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13163.904506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13163.941372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13165.861510] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13167.456902] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13167.538858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13168.882259] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13168.954538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13174.242908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13174.243750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13174.248183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13174.250136] LustreError: Skipped 1 previous similar message [13174.252582] Lustre: Skipped 1 previous similar message [13175.778465] Lustre: server umount lustre-OST0000 complete [13177.824987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13177.827413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.831220] Lustre: Skipped 1 previous similar message [13177.833438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13179.361355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13179.362062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.260394] Lustre: server umount lustre-MDT0000 complete [13184.626572] LustreError: 486416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773619857 with bad export cookie 16858166656281684928 [13184.630331] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13184.630884] LustreError: 486416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13184.752909] Lustre: server umount lustre-MDT0001 complete [13193.585947] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [13196.991632] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [13208.167635] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12872 sec ========= 20:11:20 (1773619880) [13208.763440] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:11:21 (1773619881) === [13209.954411] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:11:22 (1773619882) === [13223.569468] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [13224.575091] Key type lgssc unregistered [13224.709424] LNet: 492848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13224.713027] LNetError: 492848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13224.723534] LNet: Removed LNI 192.168.203.134@tcp [13225.034492] Key type .llcrypt unregistered [13225.035944] Key type ._llcrypt unregistered