[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 543763883 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003097] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007933] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010108] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.020240] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025512] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.031325] *** VALIDATE proc *** [ 0.032258] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036105] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038037] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042107] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.044844] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045763] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049018] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053306] rcu: Hierarchical SRCU implementation. [ 0.055520] smp: Bringing up secondary CPUs ... [ 0.056626] x86: Booting SMP configuration: [ 0.057038] .... node #0, CPUs: #1 #2 #3 [ 0.060223] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146018] node 0 deferred pages initialised in 81ms [ 0.149109] devtmpfs: initialized [ 0.150236] x86/mm: Memory block size: 128MB [ 0.153930] gcov: version magic: 0x41383552 [ 0.155167] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158243] pinctrl core: initialized pinctrl subsystem [ 0.159113] [ 0.159489] ************************************************************* [ 0.161007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162005] ** ** [ 0.163007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165007] ** ** [ 0.166006] ** This means that this kernel is built to expose internal ** [ 0.167007] ** IOMMU data structures, which may compromise security on ** [ 0.168007] ** your system. ** [ 0.170007] ** ** [ 0.171008] ** If you see this message and you are not debugging the ** [ 0.172006] ** kernel, report this immediately to your vendor! ** [ 0.174007] ** ** [ 0.176012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178010] ************************************************************* [ 0.179591] NET: Registered protocol family 16 [ 0.180324] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187029] cpuidle: using governor menu [ 0.189061] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190589] PCI: Using configuration type 1 for base access [ 0.192226] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208034] cryptd: max_cpu_qlen set to 1000 [ 0.211324] ACPI: Added _OSI(Module Device) [ 0.212014] ACPI: Added _OSI(Processor Device) [ 0.214013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221221] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227263] ACPI: Interpreter enabled [ 0.228073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230015] ACPI: Using IOAPIC for interrupt routing [ 0.232177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248236] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266530] acpiphp: Slot [2] registered [ 0.268173] acpiphp: Slot [5] registered [ 0.269226] acpiphp: Slot [6] registered [ 0.271117] acpiphp: Slot [7] registered [ 0.272185] acpiphp: Slot [8] registered [ 0.274115] acpiphp: Slot [9] registered [ 0.275108] acpiphp: Slot [10] registered [ 0.276123] acpiphp: Slot [3] registered [ 0.278072] acpiphp: Slot [4] registered [ 0.279077] acpiphp: Slot [11] registered [ 0.280066] acpiphp: Slot [12] registered [ 0.281085] acpiphp: Slot [13] registered [ 0.282112] acpiphp: Slot [14] registered [ 0.283090] acpiphp: Slot [15] registered [ 0.284080] acpiphp: Slot [16] registered [ 0.285082] acpiphp: Slot [17] registered [ 0.287072] acpiphp: Slot [18] registered [ 0.288095] acpiphp: Slot [19] registered [ 0.289092] acpiphp: Slot [20] registered [ 0.291109] acpiphp: Slot [21] registered [ 0.292068] acpiphp: Slot [22] registered [ 0.293079] acpiphp: Slot [23] registered [ 0.294081] acpiphp: Slot [24] registered [ 0.295099] acpiphp: Slot [25] registered [ 0.297108] acpiphp: Slot [26] registered [ 0.298169] acpiphp: Slot [27] registered [ 0.300158] acpiphp: Slot [28] registered [ 0.301172] acpiphp: Slot [29] registered [ 0.302108] acpiphp: Slot [30] registered [ 0.304110] acpiphp: Slot [31] registered [ 0.306068] PCI host bridge to bus 0000:00 [ 0.307023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.330228] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336598] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347806] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372484] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376730] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381836] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401634] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407314] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474909] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.484023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.513720] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.560018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.568345] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.584014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.614419] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.639021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.649266] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.657020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.688225] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.690370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.692328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694736] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.701434] iommu: Default domain type: Passthrough [ 0.702284] SCSI subsystem initialized [ 0.703093] ACPI: bus type USB registered [ 0.704157] usbcore: registered new interface driver usbfs [ 0.706050] usbcore: registered new interface driver hub [ 0.707060] usbcore: registered new device driver usb [ 0.707981] pps_core: LinuxPPS API ver. 1 registered [ 0.709009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.711047] PTP clock support registered [ 0.713021] EDAC MC: Ver: 3.0.0 [ 0.714207] PCI: Using ACPI for IRQ routing [ 0.716984] NetLabel: Initializing [ 0.718014] NetLabel: domain hash size = 128 [ 0.720012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.722084] NetLabel: unlabeled traffic allowed by default [ 0.724139] vgaarb: loaded [ 0.726248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.727015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.734000] clocksource: Switched to clocksource kvm-clock [ 0.834300] VFS: Disk quotas dquot_6.6.0 [ 0.836633] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.839159] *** VALIDATE ramfs *** [ 0.840373] *** VALIDATE hugetlbfs *** [ 0.842651] pnp: PnP ACPI init [ 0.845245] pnp: PnP ACPI: found 6 devices [ 0.862957] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.866330] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.868555] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.870826] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.873272] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.875543] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.878381] NET: Registered protocol family 2 [ 0.880519] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.885375] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888274] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893195] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.896547] TCP: Hash tables configured (established 65536 bind 65536) [ 0.898346] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900112] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.901701] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903685] NET: Registered protocol family 1 [ 0.906217] RPC: Registered named UNIX socket transport module. [ 0.908680] RPC: Registered udp transport module. [ 0.910654] RPC: Registered tcp transport module. [ 0.912578] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.915322] NET: Registered protocol family 44 [ 0.917343] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.919234] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.920776] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.922625] PCI: CLS 0 bytes, default 64 [ 0.923724] Unpacking initramfs... [ 2.260951] debug: unmapping init [mem 0xffff89b5bcc54000-0xffff89b5bffbffff] [ 2.264935] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.267286] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.270342] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.750349] Initialise system trusted keyrings [ 2.751952] Key type blacklist registered [ 2.754377] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.761678] zbud: loaded [ 2.764476] *** VALIDATE nfs *** [ 2.765464] *** VALIDATE nfs4 *** [ 2.766873] pstore: using deflate compression [ 2.770430] Platform Keyring initialized [ 2.856195] NET: Registered protocol family 38 [ 2.857602] Key type asymmetric registered [ 2.858882] Asymmetric key parser 'x509' registered [ 2.860480] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.863299] io scheduler mq-deadline registered [ 2.864607] io scheduler kyber registered [ 2.865760] io scheduler bfq registered [ 2.866832] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.868499] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.870342] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.871841] ACPI: Power Button [PWRF] [ 2.875026] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.882324] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.889624] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.894187] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.902651] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.928013] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.953991] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.957620] Non-volatile memory driver v1.3 [ 2.958599] Linux agpgart interface v0.103 [ 2.981851] virtio_blk virtio1: [vda] 147136 512-byte logical blocks (75.3 MB/71.8 MiB) [ 2.983751] vda: detected capacity change from 0 to 75333632 [ 2.995556] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.997213] vdb: detected capacity change from 0 to 1073741824 [ 3.008413] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.010164] vdc: detected capacity change from 0 to 2621440000 [ 3.020722] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.022506] vdd: detected capacity change from 0 to 2621440000 [ 3.032377] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.033960] vde: detected capacity change from 0 to 4294967296 [ 3.045395] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.047173] vdf: detected capacity change from 0 to 4294967296 [ 3.052173] libphy: Fixed MDIO Bus: probed [ 3.061300] usbcore: registered new interface driver usbserial_generic [ 3.063131] usbserial: USB Serial support registered for generic [ 3.065080] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.068543] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.070070] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.072432] mousedev: PS/2 mouse device common for all mice [ 3.074709] rtc_cmos 00:05: RTC can wake from S4 [ 3.077339] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.079170] rtc_cmos 00:05: registered as rtc0 [ 3.081576] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.083048] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.086168] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.088747] intel_pstate: CPU model not supported [ 3.095321] hid: raw HID events driver (C) Jiri Kosina [ 3.098235] usbcore: registered new interface driver usbhid [ 3.101231] usbhid: USB HID core driver [ 3.103434] drop_monitor: Initializing network drop monitor service [ 3.106499] Initializing XFRM netlink socket [ 3.108634] NET: Registered protocol family 10 [ 3.111587] Segment Routing with IPv6 [ 3.113092] NET: Registered protocol family 17 [ 3.115295] mpls_gso: MPLS GSO support [ 3.122551] RAS: Correctable Errors collector initialized. [ 3.124877] AVX version of gcm_enc/dec engaged. [ 3.126704] AES CTR mode by8 optimization enabled [ 3.201848] sched_clock: Marking stable (3201826148, 0)->(4112580498, -910754350) [ 3.205935] registered taskstats version 1 [ 3.208614] Loading compiled-in X.509 certificates [ 3.210781] zswap: loaded using pool lzo/zbud [ 3.237196] Key type big_key registered [ 3.249707] Key type encrypted registered [ 3.251415] ima: No TPM chip found, activating TPM-bypass! [ 3.253856] ima: Allocated hash algorithm: sha1 [ 3.255626] ima: No architecture policies found [ 3.257302] evm: Initialising EVM extended attributes: [ 3.258859] evm: security.selinux [ 3.260200] evm: security.ima [ 3.261176] evm: security.capability [ 3.261965] evm: HMAC attrs: 0x1 [ 3.263721] rtc_cmos 00:05: setting system clock to 2026-09-11 14:04:39 UTC (1789135479) [ 3.267735] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.270081] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.277114] Write protecting the kernel read-only data: 28672k [ 3.279346] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.281417] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.307875] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.313444] systemd[1]: Detected virtualization kvm. [ 3.314605] systemd[1]: Detected architecture x86-64. [ 3.316080] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.340575] systemd[1]: No hostname configured. [ 3.341773] systemd[1]: Set hostname to . [ 3.343209] random: systemd: uninitialized urandom read (16 bytes read) [ 3.344774] systemd[1]: Initializing machine ID from random generator. [ 3.489476] random: systemd: uninitialized urandom read (16 bytes read) [ 3.492195] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.496495] random: systemd: uninitialized urandom read (16 bytes read) [ 3.498996] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.503228] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.186312] device-mapper: uevent: version 1.0.3 [ 4.188958] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.071344] random: fast init done [ 5.180268] scsi host0: ata_piix [ 5.227245] scsi host1: ata_piix [ 5.228755] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.230485] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.242916] virtio_net virtio0 ens2: renamed from eth0 [ 9.679599] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.768553] random: crng init done [ 9.770046] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.327765] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.440415] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.691837] SELinux: Disabled at runtime. [ 11.747886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.757577] systemd[1]: Detected virtualization kvm. [ 11.760104] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.294360] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.297207] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.302922] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.307247] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.311139] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.319952] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.325730] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] [ 12.363482] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.783735] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.062604] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.203754] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.321292] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.333615] EDAC sbridge: Ver: 1.1.2 [ 14.841457] Key type dns_resolver registered [ 15.115065] NFS: Registering the id_resolver key type [ 15.117264] Key type id_resolver registered [ 15.118987] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-server login: [ 46.880269] libcfs: loading out-of-tree module taints kernel. [ 46.952571] Key type ._llcrypt registered [ 46.954831] Key type .llcrypt registered [ 47.112980] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 47.317057] hrtimer: interrupt took 2074866 ns [ 66.733788] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 69.096314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.111159] alg: No test for adler32 (adler32-zlib) [ 70.776451] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 72.118827] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 74.063184] Key type lgssc registered [ 75.899857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.789752] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.497583] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 162.533846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.587368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.121267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.171899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.246751] Lustre: lustre-MDT0000: new disk, initializing [ 164.415617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.435128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.319820] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.428265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.743822] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.827947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.837231] Lustre: Skipped 1 previous similar message [ 191.054326] Lustre: lustre-MDT0001: new disk, initializing [ 191.187574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.229788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 191.245076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.838067] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.066315] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.688582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.151561] Lustre: lustre-OST0000: new disk, initializing [ 216.154538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 216.164416] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.246608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.662758] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.317994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.327287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.445472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.754615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.947432] Lustre: lustre-OST0001: new disk, initializing [ 240.951489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.961558] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.083722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 249.978952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.997096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.125829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 250.673518] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.175555] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.829152] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.450732] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.191225] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing yml_node [ 294.230070] Lustre: DEBUG MARKER: Client: 2.17.57.127 [ 296.886850] Lustre: DEBUG MARKER: MDS: 2.17.57.127 [ 299.397798] Lustre: DEBUG MARKER: OSS: 2.17.57.127 [ 300.997071] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 10:09:35 EDT 2026 [ 319.615747] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 321.223514] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.936996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 339.940496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.956280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.497933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.498966] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.504523] Lustre: Skipped 2 previous similar messages [ 342.546941] Lustre: Skipped 2 previous similar messages [ 344.644036] Lustre: server umount lustre-MDT0000 complete [ 348.315238] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135824 with bad export cookie 2801926904203832533 [ 348.319219] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.325924] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 352.736020] LustreError: 8206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.736912] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.741344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.741352] Lustre: Skipped 1 previous similar message [ 352.746272] LustreError: 8206:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 352.766296] Lustre: Skipped 1 previous similar message [ 353.762208] LustreError: 6519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.802505] LustreError: 6519:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 354.661961] Lustre: server umount lustre-MDT0001 complete [ 358.818377] Lustre: server umount lustre-OST0000 complete [ 362.849316] Lustre: server umount lustre-OST0001 complete [ 370.061456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 379.908294] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 420.703786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.937830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 420.979817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 421.027455] Lustre: lustre-MDT0000: new disk, initializing [ 421.102568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.114259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 425.437471] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 435.730929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.851935] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 435.935754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 435.946070] Lustre: Skipped 1 previous similar message [ 436.054300] Lustre: lustre-MDT0001: new disk, initializing [ 436.178991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.256186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 436.274594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.956037] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 450.883453] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 456.539188] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.449674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.851215] Lustre: lustre-OST0000: new disk, initializing [ 463.858400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 463.869753] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.936719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.879547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.905232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.033113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.955685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 480.266332] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 484.705393] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.895477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 489.584995] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 489.781474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.614440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.616412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.625153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.625163] Lustre: Skipped 3 previous similar messages [ 496.630934] Lustre: Skipped 1 previous similar message [ 498.044392] Lustre: server umount lustre-OST0000 complete [ 502.760743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 502.771993] LustreError: Skipped 1 previous similar message [ 502.783102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.807801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 502.812273] Lustre: Skipped 1 previous similar message [ 510.871418] Lustre: server umount lustre-MDT0000 complete [ 511.973810] LustreError: 15088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.019665] LustreError: 15088:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 514.957944] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135991 with bad export cookie 2801926904203834066 [ 514.971122] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.979870] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 515.246423] Lustre: server umount lustre-MDT0001 complete [ 520.323803] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:13:13 (1789135993) [ 523.391485] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 525.272488] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:13:19 (1789135999) [ 532.602389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.277541] LustreError: 19460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.356557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.019873] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 538.594241] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.151919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.274605] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 562.638431] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 569.536500] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.774375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 586.826929] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 595.124360] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 617.446886] Lustre: server umount lustre-OST0000 complete [ 623.585815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.600937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.620466] Lustre: Skipped 2 previous similar messages [ 623.641510] Lustre: Skipped 3 previous similar messages [ 631.917384] Lustre: server umount lustre-MDT0000 complete [ 633.826487] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789136094/real 1789136094] req@ffff89b63fe2a940 x1876044596563200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789136110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 633.836253] LustreError: 19460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.902705] LustreError: 19460:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 634.847130] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789136095/real 1789136095] req@ffff89b508314780 x1876044596563456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789136111 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 637.573635] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136113 with bad export cookie 2801926904203834948 [ 637.585329] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 637.602779] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.251154] Lustre: server umount lustre-MDT0001 complete [ 652.298679] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 655.476670] Key type lgssc unregistered [ 655.753676] LNet: 23021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.771196] LNetError: 23021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.815302] LNet: Removed LNI 192.168.202.110@tcp [ 657.921321] Key type .llcrypt unregistered [ 657.925745] Key type ._llcrypt unregistered [ 682.125622] Key type ._llcrypt registered [ 682.128501] Key type .llcrypt registered [ 682.219950] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 683.342831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.399069] alg: No test for adler32 (adler32-zlib) [ 684.399967] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 684.595531] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 686.263993] Key type lgssc registered [ 687.341359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.986544] loop: module loaded [ 696.111799] loop0: detected capacity change from 0 to 204800000 [ 696.377911] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.388452] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.844634] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.611914] loop0: detected capacity change from 0 to 204800000 [ 703.814910] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.827376] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.853704] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.100250] loop0: detected capacity change from 0 to 204800000 [ 714.179151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.220586] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.535089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 715.592268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 715.719997] Lustre: lustre-MDT0000: new disk, initializing [ 715.783370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.804761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 720.643682] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 729.773186] loop1: detected capacity change from 0 to 204800000 [ 735.374953] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.863558] Lustre: lustre-OST07c6: new disk, initializing [ 735.870907] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 735.879536] Lustre: Skipped 1 previous similar message [ 735.889381] Lustre: 26211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 735.984605] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 741.537146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 741.550834] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 741.583327] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 742.967864] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 747.625384] Lustre: 25198:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 747.647967] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 747.655126] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 747.664162] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 747.681591] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 747.690372] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 760.550144] Lustre: 25199:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 760.568635] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.582812] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.600440] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 760.606417] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.614402] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.678374] Lustre: 25199:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 765.698833] Lustre: 25199:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 765.706457] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 765.722353] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.744411] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 765.754213] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.761223] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 765.783218] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.793122] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 765.804955] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 765.812561] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.824242] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.052291] Lustre: 25198:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 769.074408] Lustre: 25198:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.089886] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 769.097598] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.102813] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.118214] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.134842] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 769.143091] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.149792] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 769.166275] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.176106] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.195947] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.576194] Lustre: 25199:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 773.586511] Lustre: 25199:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 773.592106] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 773.598707] Lustre: 25199:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.613878] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.626961] Lustre: 25199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.640605] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 773.660975] Lustre: 25199:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.682490] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 773.693719] Lustre: 25199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.705610] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.721429] Lustre: 25199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.268257] Lustre: 25200:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 785.281864] Lustre: 25200:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 785.312657] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.325812] Lustre: 25200:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 785.349018] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.368412] Lustre: 25200:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 785.386815] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 785.403457] Lustre: 25200:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 785.423786] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.448418] Lustre: 25200:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 785.464813] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.480119] Lustre: 25200:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.017268] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.023338] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 834.038108] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 839.138106] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 844.264146] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 847.327225] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 847.517310] Lustre: server umount lustre-OST07c6 complete [ 857.482491] Lustre: server umount lustre-MDT0000 complete [ 880.081940] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 883.403267] Key type lgssc unregistered [ 883.946320] LNet: 28772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.965045] LNetError: 28772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.997986] LNet: Removed LNI 192.168.202.110@tcp [ 885.141852] Key type .llcrypt unregistered [ 885.143638] Key type ._llcrypt unregistered [ 905.132200] Key type ._llcrypt registered [ 905.137749] Key type .llcrypt registered [ 905.264290] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 906.544250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.828309] alg: No test for adler32 (adler32-zlib) [ 907.870334] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 908.078660] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 909.808218] Key type lgssc registered [ 910.962882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 919.663877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 919.726526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.143322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.327403] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 931.296390] LustreError: 30049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.316359] LustreError: 30049:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 933.323822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.774646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.332502] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 946.546631] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.312146] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 959.771670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.175162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.319199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 966.719927] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 974.319875] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 994.272173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.280933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.307985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.318216] Lustre: Skipped 1 previous similar message [ 994.332186] Lustre: Skipped 1 previous similar message [ 999.416099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.424696] Lustre: Skipped 1 previous similar message [ 999.965379] Lustre: server umount lustre-OST0000 complete [ 1004.515250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.517937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.546846] Lustre: Skipped 1 previous similar message [ 1009.633586] LustreError: 30044:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.659889] LustreError: 30044:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1009.686166] Lustre: server umount lustre-MDT0000 complete [ 1013.702463] LustreError: 30030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136489 with bad export cookie 7511119879912038843 [ 1013.704861] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1013.728491] LustreError: 30030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1014.157198] Lustre: server umount lustre-MDT0001 complete [ 1027.801639] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1032.325058] Key type lgssc unregistered [ 1032.744821] LNet: 33788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.749043] LNetError: 33788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.765214] LNet: Removed LNI 192.168.202.110@tcp [ 1034.138995] Key type .llcrypt unregistered [ 1034.140777] Key type ._llcrypt unregistered [ 1054.497734] Key type ._llcrypt registered [ 1054.499827] Key type .llcrypt registered [ 1054.602396] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1055.672870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.807532] alg: No test for adler32 (adler32-zlib) [ 1056.951443] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 1057.266545] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1059.007187] Key type lgssc registered [ 1060.792403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.341698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.384359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.883700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.215795] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1080.293884] LustreError: 35068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.326845] LustreError: 35068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1082.015814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.310635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1088.283460] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1094.874617] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.346426] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.948917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.278500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.444478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1116.612426] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1125.822696] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1141.218269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.235868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.248282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.274439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1143.278659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.280996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.300171] Lustre: Skipped 1 previous similar message [ 1146.495996] Lustre: server umount lustre-OST0000 complete [ 1153.512617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.514430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.532565] Lustre: Skipped 1 previous similar message [ 1158.121526] Lustre: server umount lustre-MDT0000 complete [ 1158.625652] LustreError: 36723:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.641296] LustreError: 36723:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1162.305769] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136638 with bad export cookie 6042558701869609074 [ 1162.312809] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.010431] Lustre: server umount lustre-MDT0001 complete [ 1177.724308] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1180.334307] Key type lgssc unregistered [ 1180.629316] LNet: 38629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.638249] LNetError: 38629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.656973] LNet: Removed LNI 192.168.202.110@tcp [ 1181.457936] Key type .llcrypt unregistered [ 1181.462789] Key type ._llcrypt unregistered [ 1205.807558] Key type ._llcrypt registered [ 1205.809195] Key type .llcrypt registered [ 1205.924374] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1206.787576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.836173] alg: No test for adler32 (adler32-zlib) [ 1208.132418] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 1208.354880] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1210.047187] Key type lgssc registered [ 1211.478078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.607842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.657893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.293659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.313978] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1232.871162] LustreError: 40048:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.891130] LustreError: 40048:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1237.992977] LustreError: 40049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.314200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.554523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.813786] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1255.182517] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1262.037189] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1268.550044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.850863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.148480] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1283.875971] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.467465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1301.149292] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1301.984842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1302.001257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.036961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.060962] Lustre: server umount lustre-MDT0000 complete [ 1304.035317] LustreError: 40048:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.041255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.192902] LustreError: 40043:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.217736] LustreError: 40043:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1307.664563] LustreError: 40030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136783 with bad export cookie 5516704924298430086 [ 1307.671183] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.675399] LustreError: 40030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1309.163751] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.176043] Lustre: Skipped 1 previous similar message [ 1309.195041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1311.308132] LustreError: 40045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.332662] LustreError: 40045:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1311.333427] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 1314.247652] Lustre: server umount lustre-MDT0001 complete [ 1335.988181] Lustre: server umount lustre-OST0000 complete [ 1352.742556] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1355.513499] Key type lgssc unregistered [ 1355.826616] LNet: 43934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.831972] LNetError: 43934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.913156] LNet: Removed LNI 192.168.202.110@tcp [ 1357.846597] Key type .llcrypt unregistered [ 1357.853884] Key type ._llcrypt unregistered [ 1385.741162] Key type ._llcrypt registered [ 1385.747355] Key type .llcrypt registered [ 1385.913700] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1387.557333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1387.575843] alg: No test for adler32 (adler32-zlib) [ 1388.811803] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 1389.111973] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1390.839933] Key type lgssc registered [ 1392.587822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1401.314484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1401.357981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.943531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.382456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1413.092818] LustreError: 45353:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.114142] LustreError: 45353:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1417.147856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.634396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1422.053174] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1430.196145] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.481489] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1445.516454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.023650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.058337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1452.996343] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1460.741301] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1467.497770] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1467.876861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.879219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1472.589700] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 1472.600191] Lustre: Skipped 2 previous similar messages [ 1472.931046] Lustre: server umount lustre-OST0000 complete [ 1493.476516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.478726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.508233] Lustre: Skipped 2 previous similar messages [ 1498.015828] Lustre: server umount lustre-MDT0000 complete [ 1498.602053] LustreError: 46082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.624042] LustreError: 46082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1502.639113] LustreError: 45333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136978 with bad export cookie 17130000561498288936 [ 1502.660326] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.083308] Lustre: server umount lustre-MDT0001 complete [ 1521.514804] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1524.450751] Key type lgssc unregistered [ 1524.862928] LNet: 49068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.882744] LNetError: 49068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.920106] LNet: Removed LNI 192.168.202.110@tcp [ 1525.827809] Key type .llcrypt unregistered [ 1525.832094] Key type ._llcrypt unregistered [ 1555.489889] Key type ._llcrypt registered [ 1555.494307] Key type .llcrypt registered [ 1555.684506] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1557.684349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1557.832712] alg: No test for adler32 (adler32-zlib) [ 1559.117993] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 1559.479172] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1561.279206] Key type lgssc registered [ 1562.282293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1572.083156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1572.106891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.721411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.813793] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1583.083205] LustreError: 50486:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1583.099596] LustreError: 50486:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1588.206475] LustreError: 50487:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.345819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.786633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1594.222961] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1602.872936] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1609.275197] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1617.814591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1618.518987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.542230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1624.749234] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1632.357388] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1639.031165] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1642.804825] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1655.137104] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1110h32m08s [ 1656.804507] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1660.392409] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.400657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1660.407643] Lustre: Skipped 1 previous similar message [ 1660.427090] Lustre: Skipped 2 previous similar messages [ 1661.480651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.497096] Lustre: Skipped 3 previous similar messages [ 1665.594478] Lustre: server umount lustre-MDT0000 complete [ 1665.615343] LustreError: 50483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.589940] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1671.751345] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.777553] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1677.911616] LustreError: 52869:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.939866] LustreError: 52869:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1679.411813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.533381] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.831806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.897776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1680.896601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1684.885483] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1690.128612] Lustre: DEBUG MARKER: Wait for df (36739) ... [ 1693.051823] Lustre: DEBUG MARKER: done [ 1700.323945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1700.337243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.369034] Lustre: Skipped 1 previous similar message [ 1700.381353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.396765] Lustre: Skipped 1 previous similar message [ 1701.346994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.267703] Lustre: server umount lustre-OST0000 complete [ 1711.586047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.592795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.610186] Lustre: Skipped 1 previous similar message [ 1711.627544] Lustre: Skipped 3 previous similar messages [ 1716.510598] Lustre: server umount lustre-MDT0000 complete [ 1718.764628] LustreError: 53173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.785410] LustreError: 53173:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1720.471737] LustreError: 52329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137196 with bad export cookie 16379585659527812977 [ 1720.474162] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.484342] LustreError: 52329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1721.104493] Lustre: server umount lustre-MDT0001 complete [ 1734.932941] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1738.306772] Key type lgssc unregistered [ 1738.653545] LNet: 55412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.658492] LNetError: 55412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.690100] LNet: Removed LNI 192.168.202.110@tcp [ 1740.727226] Key type .llcrypt unregistered [ 1740.729521] Key type ._llcrypt unregistered [ 1782.669269] Key type ._llcrypt registered [ 1782.671745] Key type .llcrypt registered [ 1782.780266] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 1783.882210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.040925] alg: No test for adler32 (adler32-zlib) [ 1785.167458] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 1785.416676] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 1787.071185] Key type lgssc registered [ 1788.210816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.782564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.803165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.208507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1798.247787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1798.473952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.068219] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1812.834850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.094200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1813.175685] Lustre: 57623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1813.541880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1818.990613] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1826.861291] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.822688] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1839.464350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1839.614354] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1839.792827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.383353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1846.142626] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 1854.417529] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.050456] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1110h35m36s [ 1863.777979] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1872.371823] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1876.068688] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1876.132519] LustreError: 58989:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1896.607311] Lustre: lustre-MDT0000: Client f74f26b7-df5f-4163-b9e8-5793132d528f (at 192.168.202.10@tcp) reconnecting [ 1916.199174] LustreError: 58989:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1919.482758] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1922.318966] Lustre: DEBUG MARKER: done [ 1932.256445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.263837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.294339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.791372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.794868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.817031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1937.256329] Lustre: server umount lustre-OST0000 complete [ 1942.233102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.256366] Lustre: Skipped 1 previous similar message [ 1942.262718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.682866] Lustre: server umount lustre-MDT0000 complete [ 1949.668957] LustreError: 57635:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.685093] LustreError: 57635:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1953.446342] LustreError: 57616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137429 with bad export cookie 9903851829240224906 [ 1953.451263] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.457506] LustreError: 57616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1953.874830] Lustre: server umount lustre-MDT0001 complete [ 1966.694616] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 1969.699524] Key type lgssc unregistered [ 1970.063806] LNet: 61958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.078792] LNetError: 61958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.120167] LNet: Removed LNI 192.168.202.110@tcp [ 1970.993237] Key type .llcrypt unregistered [ 1970.995930] Key type ._llcrypt unregistered [ 2024.485367] Key type ._llcrypt registered [ 2024.497409] Key type .llcrypt registered [ 2024.631158] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2026.292729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.393101] alg: No test for adler32 (adler32-zlib) [ 2027.580918] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 2027.896450] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2029.575191] Key type lgssc registered [ 2031.022580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.648734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2034.662645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.981627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2036.037647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2036.323465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.181792] Lustre: Failing over lustre-MDT0000 [ 2037.391531] Lustre: server umount lustre-MDT0000 complete [ 2043.809261] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:38:38 (1789137518) [ 2050.619048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.082083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.743295] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2064.983719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.159285] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2065.214231] Lustre: 64757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2065.757627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2071.383888] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2081.868048] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2088.943554] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2096.805069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2097.023171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2097.236735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.674256] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2105.050576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2114.341360] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2120.287695] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2130.918406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.920820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.938038] Lustre: Skipped 2 previous similar messages [ 2130.945626] Lustre: Skipped 2 previous similar messages [ 2134.041565] Lustre: server umount lustre-MDT0000 complete [ 2139.084637] LustreError: 67468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137615 with bad export cookie 9769599640573775931 [ 2139.103981] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.720305] Lustre: server umount lustre-MDT0001 complete [ 2144.789476] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2150.173168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2157.599809] Lustre: 63679:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137617/real 1789137617] req@00000000dd36ed51 x1876046647438208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.653570] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.779168] Lustre: 63679:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137620/real 1789137620] req@00000000a2f6f938 x1876046647438336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2159.839563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.655196] Lustre: 63681:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137622/real 1789137622] req@00000000bd95140f x1876046647438464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2164.319321] LustreError: 66606:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.346804] LustreError: 66606:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2164.452698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.590559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2164.767150] Lustre: 63681:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137625/real 1789137625] req@00000000d7b7aedf x1876046647438720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.489492] LustreError: 66607:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.593215] LustreError: 68530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.610873] LustreError: 68530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2170.427526] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2170.656714] Lustre: 63680:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137630/real 1789137630] req@000000008235a1a0 x1876046647438848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2172.925272] LustreError: 68528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.948148] LustreError: 68528:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2178.035508] LustreError: 68530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.051852] LustreError: 68530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2181.109446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.426507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.552191] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2187.177654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2195.011237] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2201.227562] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2207.202175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2207.215697] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.230538] Lustre: Skipped 2 previous similar messages [ 2207.231854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.247762] Lustre: Skipped 1 previous similar message [ 2207.495207] Lustre: server umount lustre-MDT0000 complete [ 2211.092646] LustreError: 67468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137687 with bad export cookie 9769599640573778206 [ 2211.095878] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.103272] LustreError: 67468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2211.418463] Lustre: server umount lustre-MDT0001 complete [ 2218.235238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2228.709324] Lustre: 63679:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137688/real 1789137688] req@00000000a2d394c8 x1876046647464576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2228.739726] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.559436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.604310] LustreError: 66606:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.773485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.918041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2243.053651] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2247.160549] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2247.174973] Lustre: Skipped 1 previous similar message [ 2253.702764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.440140] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2260.483663] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2269.931166] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.566685] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2285.043139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.048057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.072937] Lustre: Skipped 1 previous similar message [ 2285.088646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2290.145904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.148703] Lustre: Skipped 2 previous similar messages [ 2295.265384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.275701] Lustre: Skipped 2 previous similar messages [ 2298.336108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2298.618339] Lustre: server umount lustre-MDT0000 complete [ 2300.387982] LustreError: 71490:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.407359] LustreError: 71490:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 2302.744976] LustreError: 67671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137778 with bad export cookie 9769599640573779585 [ 2302.746056] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.763955] LustreError: 67671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.148262] Lustre: server umount lustre-MDT0001 complete [ 2309.634769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2313.056429] Lustre: DEBUG MARKER: files compared the same [ 2325.417844] Lustre: server umount lustre-OST0000 complete [ 2342.973499] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2346.756616] Key type lgssc unregistered [ 2347.219839] LNet: 74492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.225182] LNetError: 74492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.274056] LNet: Removed LNI 192.168.202.110@tcp [ 2348.467467] Key type .llcrypt unregistered [ 2348.470349] Key type ._llcrypt unregistered [ 2372.434828] Key type ._llcrypt registered [ 2372.439623] Key type .llcrypt registered [ 2372.573645] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2374.378764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2374.475576] alg: No test for adler32 (adler32-zlib) [ 2375.674944] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 2376.039877] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2377.825732] Key type lgssc registered [ 2379.158834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.292353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2388.306769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.992565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.323180] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2400.243263] LustreError: 75911:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.275749] LustreError: 75911:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2404.695810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.014600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2409.168749] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2416.599620] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2423.754161] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2432.241901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2432.624489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.691907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2440.316202] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug +malloc all [ 2449.701900] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2461.687645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2461.699190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.722542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.763115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2465.764659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.776743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.794574] Lustre: Skipped 1 previous similar message [ 2467.492381] Lustre: server umount lustre-OST0000 complete [ 2471.910232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.936404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.974948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.001354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.001837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.032666] Lustre: Skipped 1 previous similar message [ 2477.613392] Lustre: server umount lustre-MDT0000 complete [ 2479.010803] LustreError: 75912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.032359] LustreError: 75912:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2481.386165] LustreError: 75892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137957 with bad export cookie 10106669537949776687 [ 2481.394759] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2481.395982] LustreError: 75892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2481.732221] Lustre: server umount lustre-MDT0001 complete [ 2494.167837] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2497.114613] Key type lgssc unregistered [ 2497.446707] LNet: 79449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.451445] LNetError: 79449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.468534] LNet: Removed LNI 192.168.202.110@tcp [ 2498.535322] Key type .llcrypt unregistered [ 2498.537499] Key type ._llcrypt unregistered [ 2526.109599] Key type ._llcrypt registered [ 2526.111608] Key type .llcrypt registered [ 2526.219964] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2527.188881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.429960] alg: No test for adler32 (adler32-zlib) [ 2528.714550] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 2529.025121] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2530.743226] Key type lgssc registered [ 2532.275767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2541.940811] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2541.972919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.663457] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2562.783458] LustreError: 80844:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [ 2589.471285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2596.335881] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2604.376381] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2614.407907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.575687] Lustre: *** cfs_fail_loc=706, val=0*** [ 2615.154459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.243335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2620.030757] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2620.396579] LustreError: 80870:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.412312] LustreError: 80870:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2625.511403] LustreError: 80869:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.663176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.111239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.250316] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2643.520062] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2650.550418] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2654.697125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.711050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2654.715695] Lustre: Skipped 1 previous similar message [ 2654.723647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.730518] Lustre: Skipped 1 previous similar message [ 2659.540117] Lustre: server umount lustre-OST0000 complete [ 2664.928488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2664.936809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.941329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.951932] Lustre: Skipped 1 previous similar message [ 2664.973491] Lustre: Skipped 1 previous similar message [ 2669.497390] Lustre: server umount lustre-MDT0000 complete [ 2670.048803] LustreError: 81934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.070071] LustreError: 81934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2673.175276] LustreError: 80850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138149 with bad export cookie 7160914111832310377 [ 2673.184991] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2673.187940] LustreError: 80850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2673.629162] Lustre: server umount lustre-MDT0001 complete [ 2684.751598] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 2687.348583] Key type lgssc unregistered [ 2687.552566] LNet: 84463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2687.559392] LNetError: 84463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2687.570769] LNet: Removed LNI 192.168.202.110@tcp [ 2688.741526] Key type .llcrypt unregistered [ 2688.743380] Key type ._llcrypt unregistered [ 2715.442784] Key type ._llcrypt registered [ 2715.444511] Key type .llcrypt registered [ 2715.586654] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2717.045944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2717.077132] alg: No test for adler32 (adler32-zlib) [ 2718.236730] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 2718.507843] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 2720.183716] Key type lgssc registered [ 2721.393862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.167290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2729.183881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.676720] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2742.192211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.786514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2747.281386] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2747.872504] LustreError: 86569:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.880696] LustreError: 86569:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2751.973764] LustreError: 86573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.003041] LustreError: 86573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2754.413130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.806340] LustreError: 86569:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.859931] LustreError: 86569:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2754.889062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.161626] LustreError: 86568:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.189024] LustreError: 86568:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2760.942499] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2765.290122] LustreError: 86572:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.316703] LustreError: 86572:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2770.316766] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.331335] Lustre: MGS: Client 69f4ae2c-0359-454e-8062-ea6ae0131a97 (at 0@lo) reconnecting [ 2770.360533] LustreError: 86583:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2770.541042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.617715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2775.273752] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2790.887518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2790.894074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.903951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2791.400647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.405599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2791.435119] Lustre: Skipped 1 previous similar message [ 2794.906363] Lustre: server umount lustre-OST0000 complete [ 2801.632880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.639757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2801.650370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2801.653729] Lustre: Skipped 1 previous similar message [ 2806.367208] Lustre: server umount lustre-MDT0000 complete [ 2806.759388] LustreError: 86568:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.789398] LustreError: 86568:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2811.033451] Lustre: server umount lustre-MDT0001 complete [ 2814.711960] Lustre: server umount MGS complete [ 2826.185821] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:51:40 (1789138300) [ 2859.027314] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 2871.117892] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 2906.956278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.999765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2921.954504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.322131] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2922.409772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2922.536900] Lustre: lustre-MDT0001: new disk, initializing [ 2922.666623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.710517] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2922.725896] Lustre: 93738:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2922.742515] LustreError: 93738:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2926.991827] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2941.209193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.575614] Lustre: lustre-OST0000: new disk, initializing [ 2941.583169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2941.588286] Lustre: 94505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2941.683805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.128303] Lustre: 94790:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c05bf9f9 x1876047370952320/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789138440 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2948.620124] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2961.882538] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2968.543901] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2969.077700] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2969.085931] Lustre: MGS: Client f044e41a-fbe7-4e0d-aef9-51c5d8f3944e (at 0@lo) reconnecting [ 2969.125793] Lustre: 93006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2969.142790] LustreError: 93724:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2969.182440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2969.284248] Lustre: lustre-MDT0000: new disk, initializing [ 2969.415648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.438291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2970.480330] Lustre: 95574:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003e5af596 x1876047370959744/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789138462 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2974.004916] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 2975.525263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2975.530628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2975.692227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3001.316234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3001.318137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.324570] LustreError: Skipped 1 previous similar message [ 3001.327259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3001.355147] Lustre: Skipped 2 previous similar messages [ 3006.435709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.448922] Lustre: Skipped 2 previous similar messages [ 3006.812686] Lustre: server umount lustre-OST0000 complete [ 3011.558720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3011.562192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.588988] Lustre: Skipped 1 previous similar message [ 3016.673050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.678063] Lustre: Skipped 3 previous similar messages [ 3017.592702] Lustre: server umount lustre-MDT0000 complete [ 3027.579381] Lustre: server umount lustre-MDT0001 complete [ 3031.581343] Lustre: server umount MGS complete [ 3047.668251] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:55:21 (1789138521) [ 3069.685423] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3072.924667] Key type lgssc unregistered [ 3073.314112] LNet: 98077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3073.321355] LNetError: 98077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3073.337309] LNet: Removed LNI 192.168.202.110@tcp [ 3074.265450] Key type .llcrypt unregistered [ 3074.269094] Key type ._llcrypt unregistered [ 3096.070895] Key type ._llcrypt registered [ 3096.073683] Key type .llcrypt registered [ 3096.259984] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3098.155948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3098.233275] alg: No test for adler32 (adler32-zlib) [ 3099.350239] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 3099.660898] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3101.367244] Key type lgssc registered [ 3102.953214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3115.540287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3115.586324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.265569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.567596] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3127.780236] LustreError: 99302:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.795204] LustreError: 99302:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3132.904694] LustreError: 99301:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.741355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.106951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.208885] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3148.723440] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3155.440046] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3163.215260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.657941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.772721] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3181.538320] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3188.216801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3203.571908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.584674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.605597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.621172] Lustre: Skipped 1 previous similar message [ 3203.657632] Lustre: Skipped 1 previous similar message [ 3208.688046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.694215] Lustre: Skipped 1 previous similar message [ 3209.346138] Lustre: server umount lustre-OST0000 complete [ 3216.359043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3216.366229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.385809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.920958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.923166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.993945] Lustre: server umount lustre-MDT0000 complete [ 3224.034471] LustreError: 99301:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.060794] LustreError: 99301:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3225.691950] LustreError: 99282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138701 with bad export cookie 4224358605027910418 [ 3225.716042] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.731845] LustreError: 99282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3226.142579] Lustre: server umount lustre-MDT0001 complete [ 3240.908340] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3244.302869] Key type lgssc unregistered [ 3244.724319] LNet: 102865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.740721] LNetError: 102865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.769873] LNet: Removed LNI 192.168.202.110@tcp [ 3246.040878] Key type .llcrypt unregistered [ 3246.051099] Key type ._llcrypt unregistered [ 3268.579191] Key type ._llcrypt registered [ 3268.580885] Key type .llcrypt registered [ 3268.674841] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3270.733587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3270.773540] alg: No test for adler32 (adler32-zlib) [ 3272.031086] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 3272.503262] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3274.311203] Key type lgssc registered [ 3275.816519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3283.700326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3283.707083] LustreError: 103948:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3283.719217] LustreError: 103949:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3283.730031] LustreError: 103948:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 3283.733294] LustreError: 103948:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3283.744648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.744962] LustreError: 103948:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3285.537217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.722038] LustreError: 103986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.753034] LustreError: 103986:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3295.833520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.316558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3302.126428] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3309.641887] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing lsmod [ 3313.551693] LustreError: 105155:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3313.578263] LustreError: 105156:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3313.595095] LustreError: 105155:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3313.601456] LustreError: 105155:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3313.616512] LustreError: 105155:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3313.646679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.130805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.270520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.274613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3319.306627] Lustre: Skipped 2 previous similar messages [ 3319.311205] Lustre: Skipped 1 previous similar message [ 3321.387901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.399944] Lustre: Skipped 2 previous similar messages [ 3324.195597] Lustre: server umount lustre-MDT0000 complete [ 3326.437078] LustreError: 103983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.466209] LustreError: 103983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3328.148368] LustreError: 103962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138804 with bad export cookie 13972819082143197903 [ 3328.160913] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3329.695371] LustreError: 104418:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000477d165d x1876047951844480/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3330.110486] Lustre: server umount lustre-MDT0001 complete [ 3344.802811] Lustre: server umount lustre-OST0000 complete [ 3354.962441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.535332] LustreError: 106219:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.605134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.736856] LustreError: 106220:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.591941] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3364.836900] LustreError: 106219:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.337274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.017311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3377.324921] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3385.738097] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3392.524703] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.712053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.268864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.105128] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3420.065259] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.851549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3434.979049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3434.983781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.993024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3435.001362] Lustre: Skipped 1 previous similar message [ 3440.098123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.139215] Lustre: Skipped 1 previous similar message [ 3440.784203] Lustre: server umount lustre-OST0000 complete [ 3448.801732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3448.827275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.863099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.880170] Lustre: Skipped 1 previous similar message [ 3452.893440] Lustre: server umount lustre-MDT0000 complete [ 3455.462431] LustreError: 106219:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.492754] LustreError: 106219:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3457.094312] LustreError: 106201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138933 with bad export cookie 13972819082143198610 [ 3457.099528] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3457.107784] LustreError: 106201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3457.659383] Lustre: server umount lustre-MDT0001 complete [ 3470.945732] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3473.414297] Key type lgssc unregistered [ 3473.694700] LNet: 109793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3473.698717] LNetError: 109793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3473.728782] LNet: Removed LNI 192.168.202.110@tcp [ 3474.910884] Key type .llcrypt unregistered [ 3474.918625] Key type ._llcrypt unregistered [ 3503.646063] Key type ._llcrypt registered [ 3503.649941] Key type .llcrypt registered [ 3503.735440] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3505.576426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.651848] alg: No test for adler32 (adler32-zlib) [ 3506.890610] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 3507.155624] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3508.855360] Key type lgssc registered [ 3510.319770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3519.478595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3519.504435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.167758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.265871] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3526.629229] LustreError: 111210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.660499] LustreError: 111210:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3531.763865] LustreError: 111211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.875753] LustreError: 111210:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.091837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3538.935708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3545.909478] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3554.115768] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3561.042604] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3568.363180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3568.825924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.978507] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3584.534275] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3591.382260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3596.446792] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3600.738308] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3607.010413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3607.034439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.035706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.059769] Lustre: Skipped 1 previous similar message [ 3611.474221] Lustre: server umount lustre-OST0000 complete [ 3617.252109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.270590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.274563] Lustre: Skipped 1 previous similar message [ 3617.302337] Lustre: Skipped 2 previous similar messages [ 3622.305538] Lustre: server umount lustre-MDT0000 complete [ 3622.368419] LustreError: 111941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.385301] LustreError: 111941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3627.113028] LustreError: 111193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139103 with bad export cookie 12350430129384205317 [ 3627.114733] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.131457] LustreError: 111193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3627.508891] Lustre: server umount lustre-MDT0001 complete [ 3643.085614] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3646.990556] Key type lgssc unregistered [ 3647.369286] LNet: 114874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3647.378754] LNetError: 114874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3647.414744] LNet: Removed LNI 192.168.202.110@tcp [ 3648.616262] Key type .llcrypt unregistered [ 3648.620238] Key type ._llcrypt unregistered [ 3670.568883] Key type ._llcrypt registered [ 3670.572875] Key type .llcrypt registered [ 3670.754684] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3672.611569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3672.754547] alg: No test for adler32 (adler32-zlib) [ 3673.987671] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 3674.352756] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3676.007214] Key type lgssc registered [ 3677.479717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3687.539180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3687.568284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.094063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.971575] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3699.683222] LustreError: 116096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.702403] LustreError: 116096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3702.356749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.698509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.234647] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3718.238819] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3724.390708] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3731.263836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.647230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.732208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3738.319656] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3746.560983] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3757.541106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.552320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.562261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.572829] Lustre: Skipped 1 previous similar message [ 3757.597730] Lustre: Skipped 1 previous similar message [ 3762.671300] LustreError: 117915:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.805666] Lustre: server umount lustre-OST0000 complete [ 3769.320032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3769.337144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.362671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.899295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.914264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.439921] Lustre: server umount lustre-MDT0000 complete [ 3778.026196] LustreError: 118520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.039807] LustreError: 118520:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3778.649337] LustreError: 116077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139254 with bad export cookie 5507402609259183822 [ 3778.660923] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3778.664525] LustreError: 116077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3778.981384] Lustre: server umount lustre-MDT0001 complete [ 3790.164839] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3792.790706] Key type lgssc unregistered [ 3793.164864] LNet: 119634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3793.168159] LNetError: 119634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3793.186184] LNet: Removed LNI 192.168.202.110@tcp [ 3793.974389] Key type .llcrypt unregistered [ 3793.975803] Key type ._llcrypt unregistered [ 3819.242603] Key type ._llcrypt registered [ 3819.245767] Key type .llcrypt registered [ 3819.436547] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 3821.002423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3821.093776] alg: No test for adler32 (adler32-zlib) [ 3822.379034] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 3822.733095] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 3824.511350] Key type lgssc registered [ 3826.448136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3837.164928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3837.177564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.617040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3844.587763] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3848.164538] LustreError: 121052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.210305] LustreError: 121052:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3853.288221] LustreError: 121051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3854.626864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.184516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.751624] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3867.483947] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3874.282704] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3882.670497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3883.060988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.144663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3890.857470] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 3899.532882] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.184306] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3910.244977] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3910.259137] Lustre: Skipped 2 previous similar messages [ 3912.831202] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3912.900996] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3914.749450] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3914.756140] Lustre: Skipped 2 previous similar messages [ 3918.526391] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3918.545357] Lustre: Skipped 1 previous similar message [ 3924.410862] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.10@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3924.421276] Lustre: Skipped 2 previous similar messages [ 3926.005329] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.10@tcp 192.168.0.[2,10]@tcp [ 3931.772583] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.10/24@tcp in log lustre-MDT0000 [ 3931.781173] Lustre: Skipped 2 previous similar messages [ 3940.435925] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.10/24@tcp [ 3940.441230] Lustre: Skipped 1 previous similar message [ 3946.464950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.466339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3946.478077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.483038] Lustre: Skipped 1 previous similar message [ 3948.659964] Lustre: server umount lustre-OST0000 complete [ 3952.608362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3952.618137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.645337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3952.654197] Lustre: Skipped 1 previous similar message [ 3956.708026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.708648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.725043] Lustre: Skipped 1 previous similar message [ 3959.011810] Lustre: server umount lustre-MDT0000 complete [ 3961.828357] LustreError: 122745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.864461] LustreError: 122745:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3963.173042] LustreError: 121032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139439 with bad export cookie 13716200881615469116 [ 3963.174450] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.180896] LustreError: 121032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3963.557461] Lustre: server umount lustre-MDT0001 complete [ 3976.091993] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3979.347074] Key type lgssc unregistered [ 3979.790152] LNet: 125953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3979.805402] LNetError: 125953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3979.852610] LNet: Removed LNI 192.168.202.110@tcp [ 3980.937450] Key type .llcrypt unregistered [ 3980.939096] Key type ._llcrypt unregistered [ 4018.815588] Key type ._llcrypt registered [ 4018.817909] Key type .llcrypt registered [ 4018.969571] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4020.370172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4020.424910] alg: No test for adler32 (adler32-zlib) [ 4021.770296] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 4022.063264] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4023.847240] Key type lgssc registered [ 4024.988198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4034.299413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4034.333786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.737051] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4035.757937] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.10/24@tcp [ 4035.766410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.979477] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4045.283413] LustreError: 127519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.300037] LustreError: 127519:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4049.787817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.272704] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4050.315320] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.10/24@tcp [ 4050.406057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.443120] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4063.391317] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4068.858114] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.711288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.163172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.315927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4082.429267] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4089.292942] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4104.165421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4104.173822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4104.194638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.208859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.211357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.755418] Lustre: server umount lustre-OST0000 complete [ 4116.454711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.471722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4116.481785] Lustre: Skipped 2 previous similar messages [ 4119.801182] Lustre: server umount lustre-MDT0000 complete [ 4121.581748] LustreError: 127514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.614643] LustreError: 127514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4123.240229] LustreError: 127500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139599 with bad export cookie 14090776563950671734 [ 4123.252147] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4123.258562] LustreError: 127500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4123.717942] Lustre: server umount lustre-MDT0001 complete [ 4134.812796] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4137.560350] Key type lgssc unregistered [ 4137.851022] LNet: 131111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4137.858546] LNetError: 131111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4137.882903] LNet: Removed LNI 192.168.202.110@tcp [ 4139.009532] Key type .llcrypt unregistered [ 4139.018155] Key type ._llcrypt unregistered [ 4164.191483] Key type ._llcrypt registered [ 4164.193213] Key type .llcrypt registered [ 4164.302074] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4178.717491] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4179.759855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4179.793079] alg: No test for adler32 (adler32-zlib) [ 4180.842426] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 4181.017093] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4182.639773] Key type lgssc registered [ 4183.771913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4216.744189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4216.777045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.036516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4218.133404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4218.269796] Lustre: lustre-MDT0000: new disk, initializing [ 4218.388402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.413556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4223.854937] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4237.885246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.079968] Lustre: 134510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4238.207700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4238.219106] Lustre: Skipped 1 previous similar message [ 4238.504931] Lustre: lustre-MDT0001: new disk, initializing [ 4238.733809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4238.839637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4238.857221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4243.860384] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4253.566290] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4261.608271] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4269.236986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.486567] Lustre: lustre-OST0000: new disk, initializing [ 4269.496826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4269.508276] Lustre: 136481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4269.594613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.435868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4277.332606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4277.349599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4277.490370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4287.083500] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4291.865894] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4292.122037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4296.789974] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4297.013577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4303.329965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4303.331853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.346990] LustreError: Skipped 1 previous similar message [ 4303.357310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.370409] Lustre: Skipped 1 previous similar message [ 4303.393796] Lustre: Skipped 1 previous similar message [ 4305.258555] Lustre: server umount lustre-OST0000 complete [ 4310.496748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4310.516334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.542418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.571353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.571987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.611759] Lustre: Skipped 1 previous similar message [ 4315.527411] Lustre: server umount lustre-MDT0000 complete [ 4319.605485] LustreError: 134502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139795 with bad export cookie 2215170281548805751 [ 4319.614693] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4319.624348] LustreError: 134502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4320.250496] Lustre: server umount lustre-MDT0001 complete [ 4330.245111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.796093] LustreError: 138698:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.823028] LustreError: 138698:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4330.908968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.622387] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4336.110357] LustreError: 138699:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.224376] LustreError: 138698:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.977177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.457376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4350.388769] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4358.027840] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4364.633738] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4371.395104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4371.646677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4377.452915] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4384.434980] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4388.626431] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4388.826518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4393.117801] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4393.317658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4402.228628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4402.437108] Lustre: lustre-OST0001: new disk, initializing [ 4402.441838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4402.450895] Lustre: 142069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4402.565176] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4409.868376] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4412.040654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4412.080139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4412.191632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4419.749475] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4420.145194] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4426.797820] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4427.056275] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4435.100742] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91c5d1834800.ost_server_uuid 50 [ 4436.801075] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91c5d1834800.ost_server_uuid in IDLE state after 0 sec [ 4448.224091] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4448.227495] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.233486] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4448.236386] Lustre: Skipped 1 previous similar message [ 4448.247492] Lustre: Skipped 1 previous similar message [ 4452.711772] Lustre: server umount lustre-OST0001 complete [ 4453.348069] LustreError: 140522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.366347] LustreError: 140522:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4456.931482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4456.945854] LustreError: Skipped 1 previous similar message [ 4456.947913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.969540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4458.470592] LustreError: 140523:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.481599] LustreError: 140523:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4463.035455] Lustre: server umount lustre-OST0000 complete [ 4468.193093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4468.204476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.226056] Lustre: Skipped 1 previous similar message [ 4468.242161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.253136] Lustre: Skipped 2 previous similar messages [ 4473.232348] Lustre: server umount lustre-MDT0000 complete [ 4473.825468] LustreError: 138694:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.849308] LustreError: 138694:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4476.651832] LustreError: 138680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139952 with bad export cookie 2215170281548806633 [ 4476.657115] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.660042] LustreError: 138680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4476.936786] Lustre: server umount lustre-MDT0001 complete [ 4496.180352] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4498.955421] Key type lgssc unregistered [ 4499.204598] LNet: 145076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4499.219889] LNetError: 145076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4499.247842] LNet: Removed LNI 192.168.202.110@tcp [ 4500.077510] Key type .llcrypt unregistered [ 4500.079914] Key type ._llcrypt unregistered [ 4541.206702] Key type ._llcrypt registered [ 4541.209736] Key type .llcrypt registered [ 4541.297355] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4558.108750] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4559.591988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4559.696107] alg: No test for adler32 (adler32-zlib) [ 4560.738927] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 4560.967340] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4562.679196] Key type lgssc registered [ 4563.859597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4598.867603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4598.895449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.144987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4600.173328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4600.250879] Lustre: lustre-MDT0000: new disk, initializing [ 4600.330417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.365569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4605.163680] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4616.605637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.720262] Lustre: 149261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4616.774812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4616.779814] Lustre: Skipped 1 previous similar message [ 4616.843292] Lustre: lustre-MDT0001: new disk, initializing [ 4616.976537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4617.044470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4617.059611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4622.951167] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4632.314431] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4638.394144] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4646.024504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4646.422153] Lustre: lustre-OST0000: new disk, initializing [ 4646.427448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4646.436298] Lustre: 151228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4646.637843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.838393] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4656.262287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4656.283267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4656.455060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4666.796448] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.574066] Lustre: Failing over lustre-OST0000 [ 4670.836810] Lustre: server umount lustre-OST0000 complete [ 4687.825717] Lustre: 147478:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140148/real 1789140148] req@00000000a49436d8 x1876049303471744/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4687.857769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.352199] Lustre: 147476:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140148/real 1789140148] req@000000004c1eee9b x1876049303471872/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789140164 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4688.406395] Lustre: 147476:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4689.906442] Lustre: 147478:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140149/real 1789140149] req@00000000218388a6 x1876049303472128/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789140165 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4690.844742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.307973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.329031] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4693.344109] Lustre: 147479:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140153/real 1789140153] req@0000000034da022b x1876049303473280/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4693.370769] Lustre: 147479:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4698.173753] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4698.469810] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4698.607401] Lustre: 147479:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140158/real 1789140158] req@000000001a79d08f x1876049303474304/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4700.325527] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4700.327594] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4702.139600] Lustre: Failing over lustre-MDT0000 [ 4702.333555] Lustre: server umount lustre-MDT0000 complete [ 4703.715701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.729864] Lustre: Skipped 3 previous similar messages [ 4708.841170] LustreError: 149266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.878762] LustreError: 149266:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4710.474862] LustreError: 151235:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.960508] LustreError: 149268:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.005731] LustreError: 149268:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4719.071105] Lustre: 147477:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140179/real 1789140179] req@00000000b5a4cdfe x1876049303485312/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789140195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4719.079300] LustreError: 150419:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.104127] Lustre: 147477:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4719.106078] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4719.127150] LustreError: 150419:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4720.619231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.192490] LustreError: 149271:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.223832] LustreError: 149271:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4730.340202] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8337c96bbfcf9048 [ 4730.360520] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [ 4730.371024] Lustre: Skipped 1 previous similar message [ 4730.757549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.818794] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4730.955627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4735.982709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4736.005275] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4736.014369] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4736.066241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4746.223042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.236054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4750.580861] Lustre: server umount lustre-OST0000 complete [ 4756.451529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.452281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4756.465087] Lustre: Skipped 1 previous similar message [ 4756.466777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.466783] Lustre: Skipped 1 previous similar message [ 4761.458414] Lustre: server umount lustre-MDT0000 complete [ 4761.568899] LustreError: 149271:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.585418] LustreError: 149271:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4765.029221] LustreError: 150065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140241 with bad export cookie 9455247407304380488 [ 4765.043907] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4765.052850] LustreError: 150065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4765.475536] Lustre: server umount lustre-MDT0001 complete [ 4776.919510] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 4779.935853] Key type lgssc unregistered [ 4780.248520] LNet: 154889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4780.255403] LNetError: 154889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4780.272350] LNet: Removed LNI 192.168.202.110@tcp [ 4781.228616] Key type .llcrypt unregistered [ 4781.230392] Key type ._llcrypt unregistered [ 4820.578361] Key type ._llcrypt registered [ 4820.583584] Key type .llcrypt registered [ 4820.742283] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4822.241246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4822.408143] alg: No test for adler32 (adler32-zlib) [ 4823.612352] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 4824.020681] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 4825.807183] Key type lgssc registered [ 4827.171939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4835.876664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4835.892181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4837.308530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.643745] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4847.588154] LustreError: 156408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.616714] LustreError: 156408:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4852.705980] LustreError: 156407:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.863330] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.835559] LustreError: 156408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.509435] LustreError: 156436:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4859.521492] LustreError: 156436:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4859.536723] LustreError: 156436:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4860.005926] Lustre: server umount lustre-MDT0000 complete [ 4878.480618] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 4889.522890] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 4933.210055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.493334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4933.512039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4933.634798] Lustre: lustre-MDT0000: new disk, initializing [ 4933.733822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.763332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4939.048156] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4951.576880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.716897] Lustre: 159742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4951.817811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4951.822247] Lustre: Skipped 1 previous similar message [ 4951.926635] Lustre: lustre-MDT0001: new disk, initializing [ 4952.033742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4952.068972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4952.085167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4957.509585] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 4967.968893] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4974.331385] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4982.217492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4982.530399] Lustre: lustre-OST0000: new disk, initializing [ 4982.534107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4982.546538] Lustre: 161710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4982.658423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.535213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4989.547085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4989.682760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4990.352186] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5000.678370] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5005.605599] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5008.194455] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:28:01 (1789140481) [ 5225.447289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.464522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5225.472895] Lustre: Skipped 1 previous similar message [ 5225.492073] Lustre: Skipped 2 previous similar messages [ 5230.567928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5235.680897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5235.686906] Lustre: Skipped 4 previous similar messages [ 5238.239992] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5238.566915] Lustre: server umount lustre-MDT0000 complete [ 5240.800181] LustreError: 160547:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.822328] LustreError: 160547:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5243.063351] LustreError: 159734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140719 with bad export cookie 407979781677365860 [ 5243.072608] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5243.078151] LustreError: 159734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5245.920709] LustreError: 163484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.921341] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.949988] Lustre: Skipped 1 previous similar message [ 5245.961077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5249.818562] Lustre: server umount lustre-MDT0001 complete [ 5259.498351] Lustre: server umount lustre-OST0000 complete [ 5270.175974] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5280.205591] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5320.964496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.255965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5321.285061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5321.405283] Lustre: lustre-MDT0000: new disk, initializing [ 5321.544855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5321.568653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5328.090483] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5340.378556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5340.547130] Lustre: 166379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5340.654679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5340.661980] Lustre: Skipped 1 previous similar message [ 5340.776402] Lustre: lustre-MDT0001: new disk, initializing [ 5340.975878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5341.049991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5341.064175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5346.508275] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5358.761627] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5365.763773] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5372.394348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5372.848300] Lustre: lustre-OST0000: new disk, initializing [ 5372.857243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5372.875918] Lustre: 168351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5373.068828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.941421] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5382.723051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5382.733512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5382.862252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5390.819433] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5396.113963] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5396.327636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5401.491258] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5401.790464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5408.749175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5408.751786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.822944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5408.832731] Lustre: Skipped 2 previous similar messages [ 5410.316041] Lustre: server umount lustre-OST0000 complete [ 5417.955846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5417.967637] LustreError: Skipped 1 previous similar message [ 5417.978951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.997947] Lustre: Skipped 1 previous similar message [ 5418.003745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5422.094466] Lustre: server umount lustre-MDT0000 complete [ 5424.102365] LustreError: 166392:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.113632] LustreError: 166392:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5426.215113] LustreError: 166373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140902 with bad export cookie 407979781677524690 [ 5426.243521] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.933530] Lustre: server umount lustre-MDT0001 complete [ 5435.753570] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:35:10 (1789140910) [ 5456.071388] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5467.474163] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5505.441496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.786804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5505.824479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5505.903583] Lustre: lustre-MDT0000: new disk, initializing [ 5505.985075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.011800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5510.593906] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5521.413555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.616738] Lustre: 173109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5522.030108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5526.771690] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5536.362962] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5541.334294] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5548.061763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.313839] Lustre: lustre-OST0000: new disk, initializing [ 5548.316751] Lustre: Skipped 1 previous similar message [ 5548.319761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5548.323490] Lustre: Skipped 2 previous similar messages [ 5548.329435] Lustre: 175084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5554.259619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5554.266387] Lustre: Skipped 1 previous similar message [ 5554.269036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5554.363600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5555.204449] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5564.979183] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5575.137528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.145980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5575.158784] Lustre: Skipped 2 previous similar messages [ 5575.160356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.181138] Lustre: Skipped 3 previous similar messages [ 5578.698694] Lustre: server umount lustre-OST0000 complete [ 5583.329231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5583.333674] LustreError: Skipped 1 previous similar message [ 5583.338720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.532982] Lustre: server umount lustre-MDT0000 complete [ 5590.507248] LustreError: 173120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.525855] LustreError: 173120:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5592.711645] LustreError: 173101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141068 with bad export cookie 407979781677525572 [ 5592.749532] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5593.135753] Lustre: server umount lustre-MDT0001 complete [ 5601.605470] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:37:55 (1789141075) [ 5618.859375] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 5630.003109] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5671.198395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.359584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5671.378801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5671.475967] Lustre: lustre-MDT0000: new disk, initializing [ 5671.540600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.547334] Lustre: Skipped 2 previous similar messages [ 5671.560255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5676.829824] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5687.534038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.601428] Lustre: 179407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5687.772078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5693.257035] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5705.728356] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5716.033658] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5723.742411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5723.986863] Lustre: 181384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5730.580479] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5732.402422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5732.575312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5741.367543] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.327867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.329243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.370247] Lustre: Skipped 1 previous similar message [ 5753.378758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5753.388703] LustreError: Skipped 1 previous similar message [ 5753.417311] Lustre: Skipped 4 previous similar messages [ 5755.328631] Lustre: server umount lustre-OST0000 complete [ 5765.984371] Lustre: server umount lustre-MDT0000 complete [ 5768.673808] LustreError: 182287:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.690784] LustreError: 182287:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5769.806739] LustreError: 179399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141246 with bad export cookie 407979781677526657 [ 5769.818295] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.825266] LustreError: 179399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5770.285942] Lustre: server umount lustre-MDT0001 complete [ 5782.920620] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5785.377073] Key type lgssc unregistered [ 5785.709961] LNet: 183247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5785.741241] LNetError: 183247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5785.761264] LNet: Removed LNI 192.168.202.110@tcp [ 5786.587186] Key type .llcrypt unregistered [ 5786.589742] Key type ._llcrypt unregistered [ 5814.727140] Key type ._llcrypt registered [ 5814.729654] Key type .llcrypt registered [ 5814.854922] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5815.960025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5816.049195] alg: No test for adler32 (adler32-zlib) [ 5817.232845] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 5817.662255] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 5819.383904] Key type lgssc registered [ 5821.003666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5831.422331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5831.489451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.452701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.078797] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5842.923341] LustreError: 184665:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.956874] LustreError: 184665:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5848.043928] LustreError: 184664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.006861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.503032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5857.090264] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5865.523804] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5873.050495] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5881.551934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5882.001226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.020469] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 5897.123977] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5904.586525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5925.347232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.348590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5925.350680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5925.368804] Lustre: Skipped 1 previous similar message [ 5930.469367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5930.475222] Lustre: Skipped 2 previous similar messages [ 5931.624443] Lustre: server umount lustre-OST0000 complete [ 5939.680262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5939.688406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.702959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5940.706624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.047702] Lustre: server umount lustre-MDT0000 complete [ 5945.825879] LustreError: 186505:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5945.847693] LustreError: 186505:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5947.443433] LustreError: 184646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141423 with bad export cookie 2569568553509308358 [ 5947.471359] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.690657] Lustre: server umount lustre-MDT0001 complete [ 5961.001085] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 5964.581731] Key type lgssc unregistered [ 5965.077474] LNet: 188226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5965.097102] LNetError: 188226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5965.116363] LNet: Removed LNI 192.168.202.110@tcp [ 5966.241103] Key type .llcrypt unregistered [ 5966.243724] Key type ._llcrypt unregistered [ 5990.984179] Key type ._llcrypt registered [ 5990.986673] Key type .llcrypt registered [ 5991.082717] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 5992.193280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5992.323094] alg: No test for adler32 (adler32-zlib) [ 5993.393201] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 5993.583702] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 5995.255272] Key type lgssc registered [ 5996.472896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6004.793738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6004.830547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.715888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.639926] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6015.969531] LustreError: 189645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6016.000310] LustreError: 189645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6021.109100] LustreError: 189646:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.972642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.475442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.993583] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6038.316974] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6044.769956] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6052.505359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.937839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.360962] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6069.356312] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6076.471625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6085.748043] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 6086.627677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6086.636907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.642278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6086.659194] Lustre: Skipped 1 previous similar message [ 6086.672679] Lustre: Skipped 1 previous similar message [ 6090.826843] Lustre: lustre-OST0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 6091.463883] Lustre: server umount lustre-OST0000 complete [ 6098.416317] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91c5c6ef2000.ost_server_uuid 50 [ 6100.160154] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6114.554338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.566661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6114.574656] Lustre: Skipped 1 previous similar message [ 6119.904770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6119.920964] Lustre: Skipped 1 previous similar message [ 6120.969538] Lustre: server umount lustre-MDT0000 complete [ 6125.024655] LustreError: 190573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.055707] LustreError: 190573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6125.218793] LustreError: 189624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141601 with bad export cookie 5621819407020213138 [ 6125.220563] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.239653] LustreError: 189624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6125.619767] Lustre: server umount lustre-MDT0001 complete [ 6134.721567] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:46:48 (1789141608) [ 6141.548720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.195450] LustreError: 193585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.281901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.183796] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6147.561024] LustreError: 193586:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6157.214722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.664919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6162.583231] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6171.723845] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6177.793384] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6186.404333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.839551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6187.932857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6193.840993] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6202.012925] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6208.889825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6209.108680] Lustre: lustre-OST0001: new disk, initializing [ 6209.112805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6209.120579] Lustre: 196426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6209.216833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.569841] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6218.846594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6218.854264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6219.020803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6227.979593] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6233.056856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.064583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.088939] Lustre: Skipped 1 previous similar message [ 6233.106921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6234.083341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6237.232422] Lustre: server umount lustre-OST0000 complete [ 6239.306122] LustreError: 196891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6239.328643] LustreError: 196891:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6245.996475] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6246.283433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6252.270333] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6252.561331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6259.784147] LustreError: 196891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.822493] LustreError: 196891:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 6270.449926] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6270.453076] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.465032] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6270.491033] Lustre: Skipped 2 previous similar messages [ 6270.533643] Lustre: Skipped 4 previous similar messages [ 6274.233802] Lustre: server umount lustre-OST0001 complete [ 6280.673114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.675303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6280.694100] LustreError: Skipped 1 previous similar message [ 6285.384606] Lustre: server umount lustre-MDT0000 complete [ 6289.390124] LustreError: 193565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141765 with bad export cookie 5621819407020214209 [ 6289.409750] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6289.410178] LustreError: 193565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6289.685775] Lustre: server umount lustre-MDT0001 complete [ 6316.236335] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:49:50 (1789141790) [ 6324.239169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.603552] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6324.687512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6325.028205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.545607] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6339.089439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.204208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6339.249578] Lustre: 199871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6343.761845] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6351.046729] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6357.316270] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6365.554550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6365.881804] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6371.492682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6372.355941] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6381.096536] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6387.474978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.523422] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6394.744870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6394.817850] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6403.956629] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6407.137778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6407.146613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.156056] Lustre: Skipped 1 previous similar message [ 6407.161759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6407.168307] Lustre: Skipped 2 previous similar messages [ 6413.375635] Lustre: server umount lustre-OST0000 complete [ 6415.351684] LustreError: 201720:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.374247] LustreError: 201720:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 6433.560624] Lustre: server umount lustre-OST0001 complete [ 6440.934963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.968917] Lustre: Skipped 2 previous similar messages [ 6447.127659] Lustre: server umount lustre-MDT0000 complete [ 6451.492440] LustreError: 200615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141927 with bad export cookie 5621819407020215616 [ 6451.503628] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6451.519086] LustreError: 200615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6461.119894] Lustre: server umount lustre-MDT0001 complete [ 6487.814451] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:52:42 (1789141962) [ 6505.739721] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 6519.075648] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 6559.574486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.108540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6560.145446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6560.305406] Lustre: lustre-MDT0000: new disk, initializing [ 6560.471347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.474446] Lustre: Skipped 3 previous similar messages [ 6560.518883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6568.275784] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6580.489967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6580.635850] Lustre: 207969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6580.682700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6580.688640] Lustre: Skipped 1 previous similar message [ 6580.854471] Lustre: lustre-MDT0001: new disk, initializing [ 6581.161127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6581.182971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6585.064607] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6595.988902] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6604.270168] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6613.550677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6613.864221] Lustre: lustre-OST0000: new disk, initializing [ 6613.870280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6613.875864] Lustre: 209940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6621.930657] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6622.329769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6622.352141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6622.501071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6632.780656] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6638.677876] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6638.896722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6644.905073] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6645.179567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6648.299749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6648.313342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.324890] LustreError: Skipped 2 previous similar messages [ 6648.327396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6648.327405] Lustre: Skipped 8 previous similar messages [ 6648.381317] Lustre: Skipped 1 previous similar message [ 6653.431541] LustreError: 209925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.456049] LustreError: 209925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 6653.637915] Lustre: server umount lustre-OST0000 complete [ 6664.820394] Lustre: server umount lustre-MDT0000 complete [ 6668.835673] LustreError: 207961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142145 with bad export cookie 5621819407020217401 [ 6668.840878] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.864696] LustreError: 207961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6669.602892] Lustre: server umount lustre-MDT0001 complete [ 6679.917956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.932166] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6694.851682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.132830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6695.142393] Lustre: Skipped 3 previous similar messages [ 6700.041554] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6708.927850] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6715.588517] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.487589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6731.606859] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6741.009950] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6749.492745] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6749.924602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6755.213631] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6755.503602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6760.933213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.939568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.946982] LustreError: Skipped 2 previous similar messages [ 6760.979715] Lustre: Skipped 3 previous similar messages [ 6763.899314] Lustre: server umount lustre-OST0000 complete [ 6770.919703] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6776.475298] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6806.370605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.833223] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6820.368567] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.820707] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6826.153778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6826.172504] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6826.199477] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6831.586182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6831.588668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6831.600213] LustreError: Skipped 1 previous similar message [ 6831.611365] Lustre: Skipped 7 previous similar messages [ 6842.850598] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6843.107159] Lustre: server umount lustre-OST0000 complete [ 6861.279183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6861.702225] Lustre: server umount lustre-MDT0000 complete [ 6862.310221] LustreError: 212886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.336433] LustreError: 212886:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 6867.241902] LustreError: 212140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142343 with bad export cookie 5621819407020218276 [ 6867.251310] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.260510] LustreError: 212140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6867.724129] Lustre: server umount lustre-MDT0001 complete [ 6877.737608] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:59:12 (1789142352) [ 6884.557081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.116315] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6898.424087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.171254] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6912.907484] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6919.103348] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6925.348235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.180131] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6942.352383] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6948.385139] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6948.849775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6954.411837] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6954.858777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6962.498734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6962.784625] Lustre: lustre-OST0001: new disk, initializing [ 6962.788578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6962.796679] Lustre: 221501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6962.915204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6962.920814] Lustre: Skipped 5 previous similar messages [ 6970.704277] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 6972.062591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6972.077633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6972.198113] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6981.716366] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6988.122864] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6988.322373] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6993.539946] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6993.756735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6997.984497] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6997.993210] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.007911] Lustre: Skipped 5 previous similar messages [ 7001.486205] Lustre: server umount lustre-OST0001 complete [ 7009.022189] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7009.263929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7014.486364] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7014.656303] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7043.794158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.194673] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7049.213504] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7049.217442] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7049.231672] Lustre: Skipped 2 previous similar messages [ 7051.367535] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7058.841132] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7060.374146] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7076.323324] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.470972] Lustre: server umount lustre-OST0001 complete [ 7095.308918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7095.322661] Lustre: Skipped 19 previous similar messages [ 7100.622838] Lustre: server umount lustre-OST0000 complete [ 7119.327164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7119.719163] Lustre: server umount lustre-MDT0000 complete [ 7120.868266] LustreError: 218130:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.887617] LustreError: 218130:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 36 previous similar messages [ 7123.577506] LustreError: 218111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142599 with bad export cookie 5621819407020219249 [ 7123.583485] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.587746] LustreError: 218111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7130.326281] Lustre: server umount lustre-MDT0001 complete [ 7159.128539] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:03:52 (1789142632) [ 7168.056582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.334700] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7168.442138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7174.324762] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7183.795406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7183.905620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7183.952479] Lustre: 226763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7189.820871] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7198.131908] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7204.427482] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7212.024253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.202345] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7220.210416] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7228.683386] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7239.165512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.228075] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7246.825316] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7256.798553] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7258.887778] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7263.956324] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7264.163784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7268.871872] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7269.111527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7274.403601] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91c5d608a000.ost_server_uuid 50 [ 7276.471889] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91c5d608a000.ost_server_uuid in IDLE state after 0 sec [ 7277.574968] Lustre: Permanently deactivating lustre-OST0001 [ 7277.595240] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7282.680539] Lustre: Permanently reactivating lustre-OST0001 [ 7282.692095] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7282.695734] Lustre: Skipped 2 previous similar messages [ 7289.333165] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.356655] Lustre: Skipped 7 previous similar messages [ 7294.711325] Lustre: server umount lustre-OST0001 complete [ 7313.375528] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7313.545714] Lustre: server umount lustre-OST0000 complete [ 7324.042643] Lustre: server umount lustre-MDT0000 complete [ 7328.439149] LustreError: 229635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142804 with bad export cookie 5621819407020220537 [ 7328.440586] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.462746] LustreError: 229635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7329.007292] Lustre: server umount lustre-MDT0001 complete [ 7356.214326] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:07:09 (1789142829) [ 7365.654883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.950704] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7365.981738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7365.988335] Lustre: Skipped 2 previous similar messages [ 7372.273988] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7381.424528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.512354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7381.534509] Lustre: 233462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7386.615902] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7394.017793] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7400.429347] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7406.357919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.563393] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7406.591467] Lustre: Permanently deactivating lustre-OST0000 [ 7406.610210] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7412.952469] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7414.908162] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7420.039635] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7425.488023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7425.592696] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7431.686852] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7432.404017] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7437.761905] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.127117] Lustre: Permanently reactivating lustre-OST0000 [ 7439.156351] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7439.163274] Lustre: Skipped 2 previous similar messages [ 7442.052442] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7442.064731] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7442.082139] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7442.152138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7442.196302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7456.439843] Lustre: server umount lustre-OST0001 complete [ 7458.281881] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7458.289573] LustreError: Skipped 4 previous similar messages [ 7466.474404] Lustre: server umount lustre-OST0000 complete [ 7477.226681] Lustre: server umount lustre-MDT0000 complete [ 7481.609527] LustreError: 233455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142957 with bad export cookie 5621819407020222217 [ 7481.618147] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.626590] LustreError: 233455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7482.048387] Lustre: server umount lustre-MDT0001 complete [ 7493.393630] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7495.713308] Key type lgssc unregistered [ 7496.083782] LNet: 238353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7496.106028] LNetError: 238353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7497.126863] LNet: Removed LNI 192.168.202.110@tcp [ 7498.129128] Key type .llcrypt unregistered [ 7498.130762] Key type ._llcrypt unregistered [ 7520.470711] Key type ._llcrypt registered [ 7520.472535] Key type .llcrypt registered [ 7520.552729] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7521.426797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7521.576577] alg: No test for adler32 (adler32-zlib) [ 7522.790387] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 7523.013237] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7524.743315] Key type lgssc registered [ 7525.911254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7535.084616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7535.105929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.871251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.452349] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7546.337500] LustreError: 239821:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.367336] LustreError: 239821:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7547.381807] LustreError: 239822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.781351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.873461] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7551.893236] Lustre: 239811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7551.901779] Lustre: Permanently deactivating lustre-MDT0001 [ 7551.913275] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7552.050924] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7552.060915] LustreError: 239852:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7552.073929] LustreError: 239852:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7552.087267] LustreError: 239852:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7552.162901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7556.959740] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7565.104693] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.045474] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7577.259150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.573773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.609560] LustreError: 241663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.622234] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7581.674082] LustreError: 241663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.691483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7583.872522] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7587.820385] LustreError: 241733:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.854664] LustreError: 241733:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7591.391345] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7597.249122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7597.427144] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.672049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7602.692871] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7602.972431] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7610.283587] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7614.283691] Lustre: Permanently reactivating lustre-MDT0001 [ 7614.308613] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7614.315560] Lustre: Skipped 2 previous similar messages [ 7619.694380] Lustre: lustre-MDT0001: Client e803c69f-1a56-4ff5-860d-dc593a810f7e (at 192.168.202.10@tcp) reconnecting [ 7625.720221] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7627.372962] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7632.816717] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7633.000188] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7638.663566] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7638.958114] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7641.821986] Lustre: Permanently deactivating lustre-MDT0001 [ 7641.865410] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7641.879657] Lustre: Skipped 2 previous similar messages [ 7651.023679] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7663.831154] Lustre: server umount lustre-MDT0000 complete [ 7665.126801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7665.133614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.144756] LustreError: 239822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7665.162870] LustreError: 239822:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7666.149763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.157930] Lustre: Skipped 2 previous similar messages [ 7667.615141] LustreError: 239803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143143 with bad export cookie 662670911208476987 [ 7667.627290] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7667.629906] LustreError: 239803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7668.049120] Lustre: server umount lustre-MDT0001 complete [ 7674.775702] Lustre: server umount lustre-OST0000 complete [ 7688.329949] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:12:42 (1789143162) [ 7700.336457] Lustre: server umount lustre-OST0001 complete [ 7706.711557] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 7715.602562] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7751.932955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.257858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7752.267658] Lustre: Skipped 2 previous similar messages [ 7752.290740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7752.388252] Lustre: lustre-MDT0000: new disk, initializing [ 7752.469888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.493186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7758.210744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7771.319151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.415132] Lustre: 248101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7771.461906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7771.464973] Lustre: Skipped 1 previous similar message [ 7771.521752] Lustre: lustre-MDT0001: new disk, initializing [ 7771.593319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7771.623366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7771.630371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7775.807757] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7786.198392] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7791.370649] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7798.635820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7798.998745] Lustre: lustre-OST0000: new disk, initializing [ 7799.011206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7799.023160] Lustre: 250072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7799.153085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.570839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7804.587590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7804.728520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7806.563280] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7816.548724] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7818.511441] LustreError: 250080:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7826.571400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7826.757644] Lustre: lustre-OST0001: new disk, initializing [ 7826.761815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7826.768395] Lustre: 251188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7828.607080] LustreError: 250080:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7828.634963] LustreError: 248109:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7833.436392] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7834.668759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7834.679491] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7834.755832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7838.701793] LustreError: 248109:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7842.474928] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7844.235707] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7849.082812] Lustre: lustre-OST0001: Not available for connect from 192.168.202.10@tcp (stopping) [ 7849.953148] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.965202] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7849.974134] Lustre: Skipped 1 previous similar message [ 7849.985810] Lustre: Skipped 1 previous similar message [ 7852.259663] Lustre: server umount lustre-OST0001 complete [ 7854.154562] LustreError: 250057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7854.178237] LustreError: 250057:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7870.458929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.485190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.556052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.579100] Lustre: Skipped 2 previous similar messages [ 7880.677696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7880.703148] Lustre: Skipped 1 previous similar message [ 7884.255162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7884.416975] Lustre: server umount lustre-OST0000 complete [ 7889.376927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7889.391411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.413902] Lustre: Skipped 1 previous similar message [ 7889.423950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7894.939274] Lustre: server umount lustre-MDT0000 complete [ 7896.032610] LustreError: 249362:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7896.059704] LustreError: 249362:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 7899.727195] LustreError: 248093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143375 with bad export cookie 662670911208479801 [ 7899.729930] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7899.737018] LustreError: 248093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7906.252912] Lustre: server umount lustre-MDT0001 complete [ 7919.049241] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 7922.265599] Key type lgssc unregistered [ 7922.652883] LNet: 253226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7922.661190] LNetError: 253226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7922.676589] LNet: Removed LNI 192.168.202.110@tcp [ 7923.662805] Key type .llcrypt unregistered [ 7923.665570] Key type ._llcrypt unregistered [ 7967.433772] Key type ._llcrypt registered [ 7967.436238] Key type .llcrypt registered [ 7967.543236] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 7968.503102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7968.718991] alg: No test for adler32 (adler32-zlib) [ 7969.976451] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 7970.290233] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 7972.143857] Key type lgssc registered [ 7973.873441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7983.992700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7984.018398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.476159] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7985.514142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7985.660916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.799946] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 7998.927912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.999398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7999.031573] Lustre: 255436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7999.325702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.447330] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8012.270970] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8017.880857] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8026.571075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.956899] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8027.177037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.928828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8033.276161] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8040.838219] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8094.690898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.691482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8094.716561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8094.748049] Lustre: Skipped 1 previous similar message [ 8099.152605] Lustre: server umount lustre-OST0000 complete [ 8104.782472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8118.725187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8119.051111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.393773] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8134.294892] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8135.478169] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8135.487732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8135.487874] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8135.520669] Lustre: Skipped 1 previous similar message [ 8176.612512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8176.623311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.658356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8176.676132] Lustre: Skipped 2 previous similar messages [ 8176.707553] Lustre: Skipped 1 previous similar message [ 8181.080494] Lustre: server umount lustre-OST0000 complete [ 8186.857084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.878063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8191.970307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8191.986164] Lustre: Skipped 2 previous similar messages [ 8192.525478] Lustre: server umount lustre-MDT0000 complete [ 8196.497641] LustreError: 255427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143672 with bad export cookie 2304139266198918537 [ 8196.512120] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8196.518030] LustreError: 255427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8196.794894] Lustre: server umount lustre-MDT0001 complete [ 8209.862544] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8212.537258] Key type lgssc unregistered [ 8212.864288] LNet: 260859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8212.869423] LNetError: 260859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8213.944760] LNet: Removed LNI 192.168.202.110@tcp [ 8215.115691] Key type .llcrypt unregistered [ 8215.118769] Key type ._llcrypt unregistered [ 8243.070040] Key type ._llcrypt registered [ 8243.071397] Key type .llcrypt registered [ 8243.149827] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8244.654348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8244.841209] alg: No test for adler32 (adler32-zlib) [ 8246.211401] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 8246.607080] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8248.375305] Key type lgssc registered [ 8250.279688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8260.678606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8260.732027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.309169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8267.482274] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8271.850907] LustreError: 262283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8271.874103] LustreError: 262283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8276.969467] LustreError: 262282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.322284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.903290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8286.498379] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8294.198956] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8299.185714] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8306.059899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8306.433361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8307.452756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8313.269745] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8321.653963] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8348.643423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8348.657747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.697964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8349.665120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8349.688070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.694719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8354.833376] Lustre: server umount lustre-OST0000 complete [ 8361.953758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8361.968644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.993616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8361.996386] Lustre: Skipped 1 previous similar message [ 8365.026245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.028332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.151832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.164214] Lustre: Skipped 2 previous similar messages [ 8375.263814] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8375.632885] Lustre: server umount lustre-MDT0000 complete [ 8376.303314] LustreError: 262282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.341839] LustreError: 262282:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8379.889777] LustreError: 263020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143856 with bad export cookie 17071847836872586959 [ 8379.891903] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8379.928112] LustreError: 263020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8380.736712] Lustre: server umount lustre-MDT0001 complete [ 8394.521025] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8397.822148] Key type lgssc unregistered [ 8398.204854] LNet: 266597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8398.218503] LNetError: 266597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8398.235852] LNet: Removed LNI 192.168.202.110@tcp [ 8399.347440] Key type .llcrypt unregistered [ 8399.358557] Key type ._llcrypt unregistered [ 8420.181836] Key type ._llcrypt registered [ 8420.187943] Key type .llcrypt registered [ 8420.417239] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8421.979316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8422.129528] alg: No test for adler32 (adler32-zlib) [ 8423.308320] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 8423.676671] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8425.455278] Key type lgssc registered [ 8427.347643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8438.199673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8438.237809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.260565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.931961] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8450.534703] LustreError: 267821:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.572450] LustreError: 267821:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8455.582076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.654933] LustreError: 267822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.196986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8461.228690] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8469.486403] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8475.241563] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8483.155239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.641921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.699488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8490.736514] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8499.597072] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8516.582411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.594618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8516.613672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8516.629385] Lustre: Skipped 1 previous similar message [ 8518.954992] Lustre: server umount lustre-OST0000 complete [ 8526.831993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.836288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8526.845115] Lustre: Skipped 2 previous similar messages [ 8526.860852] Lustre: Skipped 1 previous similar message [ 8529.585096] Lustre: server umount lustre-MDT0000 complete [ 8531.937175] LustreError: 267821:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.954267] LustreError: 267821:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8534.144697] LustreError: 268557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144010 with bad export cookie 9156875491276934374 [ 8534.150107] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8534.159305] LustreError: 268557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8534.590524] Lustre: server umount lustre-MDT0001 complete [ 8550.757658] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8553.594612] Key type lgssc unregistered [ 8554.003970] LNet: 271505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8554.021302] LNetError: 271505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.041167] LNet: Removed LNI 192.168.202.110@tcp [ 8555.591126] Key type .llcrypt unregistered [ 8555.604940] Key type ._llcrypt unregistered [ 8584.855440] Key type ._llcrypt registered [ 8584.859037] Key type .llcrypt registered [ 8584.936395] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8586.083313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8586.134938] alg: No test for adler32 (adler32-zlib) [ 8587.220915] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 8587.538673] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8589.314822] Key type lgssc registered [ 8590.638520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8599.027914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8599.053746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.450090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.527802] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8610.788804] LustreError: 272927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.804703] LustreError: 272927:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8613.763245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.135839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8618.922616] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8626.651857] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8632.617208] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8639.292149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8639.565810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.002284] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8646.769041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8652.630253] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8679.908810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8679.929814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.976388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8680.017909] Lustre: Skipped 1 previous similar message [ 8685.060517] Lustre: server umount lustre-OST0000 complete [ 8691.173507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8691.199148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.228549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8691.231317] Lustre: Skipped 1 previous similar message [ 8695.264858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.266925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8695.310200] Lustre: Skipped 1 previous similar message [ 8697.509711] Lustre: server umount lustre-MDT0000 complete [ 8700.396267] LustreError: 275416:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.406364] LustreError: 275416:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8701.290283] LustreError: 272952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144177 with bad export cookie 11716457472030164066 [ 8701.299187] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8701.302623] LustreError: 272952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8701.679797] Lustre: server umount lustre-MDT0001 complete [ 8712.886116] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8715.417834] Key type lgssc unregistered [ 8715.818908] LNet: 277236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8715.841610] LNetError: 277236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8715.865908] LNet: Removed LNI 192.168.202.110@tcp [ 8716.593522] Key type .llcrypt unregistered [ 8716.597473] Key type ._llcrypt unregistered [ 8732.977589] Key type ._llcrypt registered [ 8732.979558] Key type .llcrypt registered [ 8733.126405] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8735.143471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8735.205158] alg: No test for adler32 (adler32-zlib) [ 8736.289475] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 8736.597569] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8738.343279] Key type lgssc registered [ 8739.334621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8746.470547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8746.486684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.086167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8752.321037] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8758.246354] LustreError: 278478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8758.264906] LustreError: 278478:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8760.092868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.551430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8766.121259] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8774.616292] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8782.036338] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8790.693988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8791.018804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8793.083743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8796.468356] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8804.975250] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8815.588763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.590650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8815.617341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8817.084486] Lustre: server umount lustre-OST0000 complete [ 8822.240656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8822.252963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8822.266425] Lustre: Skipped 1 previous similar message [ 8822.278443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.287433] Lustre: Skipped 1 previous similar message [ 8825.825956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8825.826813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8825.843824] Lustre: Skipped 1 previous similar message [ 8827.161747] Lustre: server umount lustre-MDT0000 complete [ 8830.956638] LustreError: 278482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.972834] LustreError: 278482:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8830.993730] LustreError: 278444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144307 with bad export cookie 13856635245451366310 [ 8831.013279] LustreError: 278444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8831.016565] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8831.581335] Lustre: server umount lustre-MDT0001 complete [ 8842.734881] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 8846.094211] Key type lgssc unregistered [ 8846.433703] LNet: 282182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8846.438837] LNetError: 282182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8846.465099] LNet: Removed LNI 192.168.202.110@tcp [ 8847.416783] Key type .llcrypt unregistered [ 8847.421338] Key type ._llcrypt unregistered [ 8858.368860] Key type ._llcrypt registered [ 8858.371451] Key type .llcrypt registered [ 8858.503668] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8874.884922] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 8889.156732] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 8890.293571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8890.410900] alg: No test for adler32 (adler32-zlib) [ 8891.472476] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 8891.699590] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 8893.383158] Key type lgssc registered [ 8894.376305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8929.483117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8929.531945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.892380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8930.928670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8931.004240] Lustre: lustre-MDT0000: new disk, initializing [ 8931.103673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.138421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8935.136188] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8946.492421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.626402] Lustre: 285738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8946.695104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8946.700065] Lustre: Skipped 1 previous similar message [ 8946.785700] Lustre: lustre-MDT0001: new disk, initializing [ 8946.879791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8946.929328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8946.943784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8951.968922] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8961.513715] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8966.545619] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8973.031661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8973.255597] Lustre: lustre-OST0000: new disk, initializing [ 8973.259576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8973.265509] Lustre: 287705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8973.355319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8980.815418] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 8981.114556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8981.124991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8981.427091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8990.768583] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8995.250440] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8995.498562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9000.321426] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9000.481700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9007.075616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.086071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9007.094391] Lustre: Skipped 1 previous similar message [ 9007.110352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9007.111311] Lustre: Skipped 1 previous similar message [ 9008.539760] Lustre: server umount lustre-OST0000 complete [ 9013.728834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9013.736550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.749677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.314786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.316921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9018.829864] Lustre: server umount lustre-MDT0000 complete [ 9023.270385] LustreError: 286531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144499 with bad export cookie 3613698549412130973 [ 9023.272783] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9023.280621] LustreError: 286531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9023.593620] Lustre: server umount lustre-MDT0001 complete [ 9034.527258] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:35:07 (1789144507) [ 9041.613797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.196152] LustreError: 290117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.216761] LustreError: 290117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9042.283617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9047.153895] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9047.521254] LustreError: 290118:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9052.645213] LustreError: 290117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9056.315187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9056.602321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9061.312461] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9069.710772] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9075.962932] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9082.298412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9082.642529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.008483] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9098.211586] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9113.574686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9113.587067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.614438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9113.617101] Lustre: Skipped 1 previous similar message [ 9116.641495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9118.604122] Lustre: server umount lustre-OST0000 complete [ 9123.299741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9123.312716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.338432] Lustre: Skipped 1 previous similar message [ 9123.347940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9123.353961] Lustre: Skipped 2 previous similar messages [ 9129.397261] Lustre: server umount lustre-MDT0000 complete [ 9132.018225] LustreError: 290117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.038031] LustreError: 290117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9133.739940] LustreError: 290098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144609 with bad export cookie 3613698549412131855 [ 9133.753828] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9133.763444] LustreError: 290098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9134.591646] Lustre: server umount lustre-MDT0001 complete [ 9146.808623] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9149.976425] Key type lgssc unregistered [ 9150.373540] LNet: 293680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9150.383052] LNetError: 293680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9150.416425] LNet: Removed LNI 192.168.202.110@tcp [ 9151.269341] Key type .llcrypt unregistered [ 9151.272984] Key type ._llcrypt unregistered [ 9177.612389] Key type ._llcrypt registered [ 9177.614972] Key type .llcrypt registered [ 9177.720893] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9194.925578] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9196.770907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9196.803409] alg: No test for adler32 (adler32-zlib) [ 9197.861359] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 9198.179291] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9199.863233] Key type lgssc registered [ 9201.097883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9248.128613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9248.169598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.495589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9249.525794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9249.637881] Lustre: lustre-MDT0000: new disk, initializing [ 9249.727199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.749501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9255.665195] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9267.539586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.765694] Lustre: 297523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9267.861279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9267.868749] Lustre: Skipped 1 previous similar message [ 9267.988434] Lustre: lustre-MDT0001: new disk, initializing [ 9268.161338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9268.218938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9268.238786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9273.187581] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9283.689089] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9290.979475] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9298.654399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9299.009367] Lustre: lustre-OST2710: new disk, initializing [ 9299.017211] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9299.026307] Lustre: 299493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9299.185758] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9306.884487] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9308.796982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9308.812098] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9309.000689] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9318.208462] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9325.175325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9325.360344] Lustre: lustre-OST03e8: new disk, initializing [ 9325.367164] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9325.372733] Lustre: 300556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9325.474390] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9332.850463] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9333.897073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9333.917543] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9334.047441] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9342.441612] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9351.221641] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9351.442994] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9356.030856] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9356.472591] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9361.690744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9361.882778] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9366.818316] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9367.061810] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9373.781676] Lustre: server umount lustre-MDT0000 complete [ 9375.206834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.243326] Lustre: Skipped 1 previous similar message [ 9377.533245] LustreError: 297516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144853 with bad export cookie 8729136132073842589 [ 9377.550295] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9377.554561] LustreError: 297516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9377.979657] Lustre: server umount lustre-MDT0001 complete [ 9392.518209] Lustre: server umount lustre-OST2710 complete [ 9406.706239] Lustre: server umount lustre-OST03e8 complete [ 9426.853934] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 9437.715961] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9471.975513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.219076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9472.243960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9472.303683] Lustre: lustre-MDT0000: new disk, initializing [ 9472.363171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9472.379764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9477.172880] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9489.155582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.292424] Lustre: 305808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9489.358785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9489.362880] Lustre: Skipped 1 previous similar message [ 9489.471773] Lustre: lustre-MDT0001: new disk, initializing [ 9489.578864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.612355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9489.628349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9494.249664] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9503.071250] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9507.522545] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9513.718425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9513.913228] Lustre: lustre-OST0000: new disk, initializing [ 9513.915346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9513.918419] Lustre: 307774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9513.975141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9519.237549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9519.259844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9519.385168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9520.138876] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9529.387454] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9533.926963] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9534.104779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9538.561822] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9538.821884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9545.184721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9545.188348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9545.188482] LustreError: Skipped 1 previous similar message [ 9545.192095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9545.241143] Lustre: Skipped 3 previous similar messages [ 9547.044874] Lustre: server umount lustre-OST0000 complete [ 9551.347790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9551.362333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9551.396411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9551.414362] Lustre: Skipped 1 previous similar message [ 9555.424879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9555.427058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.890639] Lustre: server umount lustre-MDT0000 complete [ 9560.553134] LustreError: 305819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9560.589927] LustreError: 305819:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9561.558050] LustreError: 305798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145037 with bad export cookie 8729136132073844269 [ 9561.565257] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9561.569829] LustreError: 305798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9561.863884] Lustre: server umount lustre-MDT0001 complete [ 9575.202688] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:44:08 (1789145048) [ 9577.344941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9579.252468] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:44:13 (1789145053) [ 9582.992960] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9612.796255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9612.958382] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9612.971609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9613.085382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.989505] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9626.003310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.091343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9626.123692] Lustre: 311631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9626.229068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9626.234236] Lustre: Skipped 1 previous similar message [ 9630.373169] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9652.329164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9652.555662] Lustre: Denying initial registration attempt from nid 192.168.202.110@tcp, specified as failover [ 9652.567364] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9652.580644] LustreError: 313083:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9652.587361] LustreError: 313083:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9652.600189] LustreError: 313083:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9652.679541] Lustre: server umount lustre-OST0000 complete [ 9652.684975] LustreError: 313083:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9662.444506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.455425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9662.457130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9665.491346] Lustre: server umount lustre-MDT0000 complete [ 9667.555648] LustreError: 312392:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.585903] LustreError: 312392:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9668.933184] LustreError: 311626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145145 with bad export cookie 8729136132073845151 [ 9668.948442] LustreError: 311626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9668.950065] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9669.214105] Lustre: server umount lustre-MDT0001 complete [ 9680.461724] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9683.727981] Key type lgssc unregistered [ 9684.060636] LNet: 314208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9684.067191] LNetError: 314208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9684.080172] LNet: Removed LNI 192.168.202.110@tcp [ 9685.214920] Key type .llcrypt unregistered [ 9685.217281] Key type ._llcrypt unregistered [ 9696.537800] Key type ._llcrypt registered [ 9696.540423] Key type .llcrypt registered [ 9696.662920] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9697.578273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9697.838982] alg: No test for adler32 (adler32-zlib) [ 9698.822484] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 9698.993746] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9700.655187] Key type lgssc registered [ 9701.643874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9746.425868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9754.900588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9754.931000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.198345] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9756.232250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9756.388660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9761.167859] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9769.546876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.725629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9769.758188] Lustre: 316972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9770.119571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9774.819604] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9782.644739] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9787.686290] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9807.338469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9807.688162] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9807.962333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.952711] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9823.820611] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9827.809419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9827.812451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9827.823747] LustreError: Skipped 1 previous similar message [ 9827.829692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9827.871043] Lustre: Skipped 1 previous similar message [ 9832.955061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9832.966516] Lustre: Skipped 1 previous similar message [ 9833.702361] Lustre: server umount lustre-OST0000 complete [ 9841.634360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9841.650398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.671455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.681244] Lustre: Skipped 1 previous similar message [ 9843.179912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.520847] Lustre: server umount lustre-MDT0000 complete [ 9849.439166] LustreError: 316964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145325 with bad export cookie 12361828357889007886 [ 9849.448795] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9849.452824] LustreError: 316964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9849.966447] Lustre: server umount lustre-MDT0001 complete [ 9864.629781] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 9867.856884] Key type lgssc unregistered [ 9868.377757] LNet: 320547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9868.381471] LNetError: 320547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9868.400767] LNet: Removed LNI 192.168.202.110@tcp [ 9869.943770] Key type .llcrypt unregistered [ 9869.948616] Key type ._llcrypt unregistered [ 9903.618805] Key type ._llcrypt registered [ 9903.621618] Key type .llcrypt registered [ 9903.787361] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 9905.351086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9905.418090] alg: No test for adler32 (adler32-zlib) [ 9906.495460] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [ 9906.692839] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 9908.328042] Key type lgssc registered [ 9909.597490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9917.772210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9917.815541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.386501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9923.501620] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9929.704453] LustreError: 321969:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.723729] LustreError: 321969:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9932.781674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.203235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9939.102765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9947.222505] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9954.354432] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9965.223848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9965.528747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9971.766656] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [ 9978.735762] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.866088] Lustre: server umount lustre-MDT0000 complete [ 9999.329867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.350026] LustreError: 321964:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.375052] LustreError: 321964:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10000.011520] LustreError: 321949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145476 with bad export cookie 3232586722852676831 [10000.030590] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10002.461082] Lustre: server umount lustre-MDT0001 complete [10008.672595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10016.856408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10019.807337] Lustre: 321281:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789145480/real 1789145480] req@000000001a95d0af x1876054908112768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789145496 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10019.844254] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.861799] Lustre: Skipped 2 previous similar messages [10020.897252] Lustre: 321281:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789145481/real 1789145481] req@00000000c8bfa8be x1876054908112896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789145497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10024.927452] Lustre: 321279:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789145485/real 1789145485] req@00000000c6efe67a x1876054908113024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789145501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10025.384448] LustreError: 323804:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.410474] LustreError: 323804:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10025.487957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.546861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10030.564945] LustreError: 323805:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.569704] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10030.935162] Lustre: 321280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789145491/real 1789145491] req@00000000d3b5abab x1876054908113408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789145507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10030.993122] Lustre: 321280:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10035.104110] Lustre: 321280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789145495/real 1789145495] req@000000008505727c x1876054908113664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789145511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10036.304516] LustreError: 325584:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.332543] LustreError: 325584:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10039.197553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.777223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10040.826612] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10044.407710] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10052.039585] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10054.294536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10061.055591] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10063.309834] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10070.497069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10070.503408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.525806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.519966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10071.523476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.533623] Lustre: Skipped 1 previous similar message [10076.649954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10081.765087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10081.770959] Lustre: Skipped 2 previous similar messages [10084.319310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10084.469833] Lustre: server umount lustre-OST0000 complete [10090.984462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10091.005714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.027570] Lustre: Skipped 1 previous similar message [10091.033543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10094.572249] Lustre: server umount lustre-MDT0000 complete [10097.122398] LustreError: 325589:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.147134] LustreError: 325589:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10098.019927] LustreError: 321947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145574 with bad export cookie 3232586722852684083 [10098.021919] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.030196] LustreError: 321947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10098.259681] Lustre: server umount lustre-MDT0001 complete [10110.552535] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10113.923314] Key type lgssc unregistered [10114.212759] LNet: 328263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10114.233199] LNetError: 328263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10114.264854] LNet: Removed LNI 192.168.202.110@tcp [10115.230395] Key type .llcrypt unregistered [10115.241151] Key type ._llcrypt unregistered [10140.887492] Key type ._llcrypt registered [10140.889944] Key type .llcrypt registered [10140.979640] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10142.404529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10142.430809] alg: No test for adler32 (adler32-zlib) [10143.541985] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [10143.776951] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10145.447180] Key type lgssc registered [10146.777594] Lustre: Echo OBD driver; http://www.lustre.org/ [10155.132251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10155.160211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10156.566921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10161.314777] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10166.760591] LustreError: 329681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10166.785451] LustreError: 329681:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10169.768700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10170.274274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10174.941929] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10183.428244] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10189.601914] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10195.107826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10195.132650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10195.136176] Lustre: Skipped 1 previous similar message [10199.470750] Lustre: server umount lustre-MDT0000 complete [10200.544784] LustreError: 329682:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.562024] LustreError: 329682:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10202.669586] LustreError: 329662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145678 with bad export cookie 14621849603783566413 [10202.675433] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10202.679814] LustreError: 329662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10203.116701] Lustre: server umount lustre-MDT0001 complete [10218.664068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.030609] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10219.083647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10219.377322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10224.637308] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10235.313745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.378632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10235.404237] Lustre: 332269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10235.742423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10240.809914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10250.078600] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10255.980470] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10266.604352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.792555] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10266.801695] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10266.811380] LustreError: 334153:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10266.819193] LustreError: 334153:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10266.821586] LustreError: 334153:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10266.943610] Lustre: server umount lustre-OST0000 complete [10266.951176] LustreError: 334153:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10278.183687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10278.354472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10278.560453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10284.023892] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10285.242921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10291.022478] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10300.104916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10300.233862] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10300.350348] Lustre: lustre-OST0001: new disk, initializing [10300.354208] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10300.359630] Lustre: 335697:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10300.440088] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10306.181779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10306.194156] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10306.293045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10306.898738] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10313.688765] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10316.778469] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10316.782236] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10316.786074] LustreError: Skipped 1 previous similar message [10316.789333] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10316.797194] Lustre: Skipped 2 previous similar messages [10316.818017] Lustre: Skipped 1 previous similar message [10321.891447] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10321.903200] Lustre: Skipped 1 previous similar message [10322.468904] Lustre: server umount lustre-OST0001 complete [10327.020364] LustreError: 334622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.028225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.058252] LustreError: 334622:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10327.085956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10327.093613] Lustre: Skipped 1 previous similar message [10332.133917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10332.134694] LustreError: 334621:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.143781] Lustre: Skipped 1 previous similar message [10332.169768] LustreError: 334621:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10332.903959] Lustre: server umount lustre-OST0000 complete [10337.254259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10337.277438] Lustre: Skipped 2 previous similar messages [10342.375667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10342.386553] Lustre: Skipped 3 previous similar messages [10342.916879] Lustre: server umount lustre-MDT0000 complete [10346.962505] LustreError: 334637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145823 with bad export cookie 14621849603783566966 [10346.974039] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10346.982546] LustreError: 334637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10347.205948] Lustre: server umount lustre-MDT0001 complete [10359.475169] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10364.180146] Key type lgssc unregistered [10364.570850] LNet: 337661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10364.581260] LNetError: 337661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10364.592634] LNet: Removed LNI 192.168.202.110@tcp [10365.889158] Key type .llcrypt unregistered [10365.891659] Key type ._llcrypt unregistered [10445.029032] Key type ._llcrypt registered [10445.033073] Key type .llcrypt registered [10445.132825] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [10462.570502] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10464.176025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10464.219960] alg: No test for adler32 (adler32-zlib) [10465.273693] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [10465.481171] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10467.167385] Key type lgssc registered [10468.568681] Lustre: Echo OBD driver; http://www.lustre.org/ [10502.723872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10502.740963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.016306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10504.052602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10504.143535] Lustre: lustre-MDT0000: new disk, initializing [10504.215230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10504.232099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10508.745595] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10519.334954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.448131] Lustre: 342601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10519.523613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10519.533316] Lustre: Skipped 1 previous similar message [10519.556251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10519.608622] Lustre: lustre-MDT0001: new disk, initializing [10519.674173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10519.737611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10519.754334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10524.616940] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10534.793981] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10539.925702] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10546.321155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.601714] Lustre: lustre-OST0000: new disk, initializing [10546.605195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10546.611076] Lustre: 344569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10546.692649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10551.625862] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10555.963448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10555.976915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10556.099386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10560.177908] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10564.311095] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10564.473582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10568.392809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10568.662566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10571.749103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10571.750609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.760844] LustreError: Skipped 1 previous similar message [10571.762494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.785722] Lustre: Skipped 1 previous similar message [10576.541323] Lustre: server umount lustre-OST0000 complete [10581.472575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10581.478734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.507098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.514175] Lustre: Skipped 1 previous similar message [10587.110175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.123738] Lustre: Skipped 2 previous similar messages [10592.225380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10592.239548] Lustre: Skipped 1 previous similar message [10594.271445] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10594.473490] Lustre: server umount lustre-MDT0000 complete [10598.149558] LustreError: 342591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146074 with bad export cookie 9917340138917929362 [10598.156898] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10598.174107] LustreError: 342591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10598.696743] Lustre: server umount lustre-MDT0001 complete [10606.972454] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:01:21 (1789146081) [10614.075935] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:01:28 (1789146088) [10623.222725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10623.826158] LustreError: 347335:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.852508] LustreError: 347335:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10623.929516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10628.904368] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10629.106284] LustreError: 347336:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.210262] LustreError: 347335:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.047321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.513113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10642.696727] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10650.797025] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10656.834489] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10663.659766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10663.923693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10671.109832] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10678.596676] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10689.075789] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10690.774695] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10692.239173] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10693.664335] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10695.203679] Lustre: Failing over lustre-MDT0000 [10695.383582] Lustre: server umount lustre-MDT0000 complete [10695.796032] LustreError: 349173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.696418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10697.715300] Lustre: Skipped 2 previous similar messages [10700.875154] LustreError: 349173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.902174] LustreError: 349173:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10711.124193] LustreError: 347331:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.140348] LustreError: 347331:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [10713.055714] Lustre: 340824:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789146173/real 1789146173] req@0000000025e1d48e x1876055494299648/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789146189 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10713.079352] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10714.290045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.317184] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x89a1784fe0ed291e [10723.331177] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [10723.637790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10723.683365] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10726.488497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10728.951173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10729.001134] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10729.063957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10729.340909] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10738.689893] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10740.086830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10743.335828] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10744.988446] Lustre: Failing over lustre-MDT0000 [10745.206615] Lustre: server umount lustre-MDT0000 complete [10746.976404] LustreError: 347330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10746.990285] LustreError: 347330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [10749.414067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10749.414584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.438601] Lustre: Skipped 3 previous similar messages [10763.163374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10763.239562] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10763.459713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10763.509215] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10767.863673] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10768.459698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10768.894944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10768.901928] Lustre: Skipped 2 previous similar messages [10768.944454] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10768.993820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10777.336310] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10779.015778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10789.345087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10789.351395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10789.371384] Lustre: Skipped 2 previous similar messages [10789.392273] Lustre: Skipped 2 previous similar messages [10791.443624] Lustre: server umount lustre-MDT0000 complete [10794.466845] LustreError: 349173:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10794.483953] LustreError: 349173:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [10794.782927] LustreError: 347317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146271 with bad export cookie 9917340138917932071 [10794.785358] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10794.796529] LustreError: 347317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10795.148577] Lustre: server umount lustre-MDT0001 complete [10808.671674] Lustre: server umount lustre-OST0000 complete [10828.660716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.631981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10835.315882] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10843.380625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.275852] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10858.328325] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10864.745532] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10873.125831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10874.503906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10880.367765] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10888.015366] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10899.432326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10899.442885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10899.446629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10899.460826] Lustre: Skipped 1 previous similar message [10899.478509] Lustre: Skipped 1 previous similar message [10901.555222] Lustre: server umount lustre-OST0000 complete [10905.573254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10911.763137] Lustre: server umount lustre-MDT0000 complete [10914.785305] LustreError: 353678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.805250] LustreError: 353678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10915.570264] LustreError: 356465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146391 with bad export cookie 9917340138917932813 [10915.574265] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10915.587764] LustreError: 356465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10915.972225] Lustre: server umount lustre-MDT0001 complete [10929.360205] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [10932.131224] Key type lgssc unregistered [10932.393331] LNet: 357221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10932.419918] LNetError: 357221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10933.477238] LNet: Removed LNI 192.168.202.110@tcp [10934.490929] Key type .llcrypt unregistered [10934.497747] Key type ._llcrypt unregistered [10961.586109] Key type ._llcrypt registered [10961.587887] Key type .llcrypt registered [10961.677465] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [10962.619719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10962.638589] alg: No test for adler32 (adler32-zlib) [10963.848441] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [10964.205967] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [10965.904179] Key type lgssc registered [10967.232601] Lustre: Echo OBD driver; http://www.lustre.org/ [10974.927716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10974.940707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.665443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10981.430533] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [10985.964846] LustreError: 358738:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10985.994712] LustreError: 358738:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10990.190856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10990.436008] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10995.181992] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11002.019865] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11007.694233] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11013.743050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11014.074197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11020.392766] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11021.247079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11028.404633] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11041.767414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.777197] LustreError: 361230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.784947] Lustre: Skipped 2 previous similar messages [11041.785711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11041.799154] LustreError: 361230:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11041.878923] Lustre: server umount lustre-MDT0000 complete [11046.371889] LustreError: 358720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146522 with bad export cookie 4748622897271650723 [11046.376746] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11046.380341] LustreError: 358720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11046.690868] Lustre: server umount lustre-MDT0001 complete [11060.017682] Lustre: server umount lustre-OST0000 complete [11075.179706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11075.716438] LustreError: 362451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.825541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11080.506545] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11081.188196] LustreError: 362452:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11085.286811] LustreError: 362451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.886819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.281251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11092.574819] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11101.110756] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11107.726437] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11114.668958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11115.110975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11120.235092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11121.674349] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11129.036535] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11144.168213] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.182892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.192770] Lustre: Skipped 1 previous similar message [11144.210054] Lustre: Skipped 2 previous similar messages [11146.403123] Lustre: server umount lustre-MDT0000 complete [11149.292483] LustreError: 362448:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.319511] LustreError: 362448:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11150.207919] LustreError: 362431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146626 with bad export cookie 4748622897271651836 [11150.223206] LustreError: 362431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11150.226370] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11150.504491] Lustre: server umount lustre-MDT0001 complete [11164.684237] Lustre: server umount lustre-OST0000 complete [11181.705326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.278320] LustreError: 366363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.407564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11187.048611] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11194.675755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.420529] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11207.745638] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11213.656228] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11221.600939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11222.049592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11222.058982] Lustre: Skipped 1 previous similar message [11227.182147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11227.186137] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11232.886443] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11237.347736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.355125] Lustre: Skipped 1 previous similar message [11237.366864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11240.426078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11242.118455] Lustre: server umount lustre-OST0000 complete [11245.537049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.537897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11245.548995] Lustre: Skipped 1 previous similar message [11245.561477] Lustre: Skipped 1 previous similar message [11250.661783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11250.668687] Lustre: Skipped 2 previous similar messages [11259.370142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11259.630203] Lustre: server umount lustre-MDT0000 complete [11260.896985] LustreError: 366359:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11260.904965] LustreError: 366359:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11262.545127] LustreError: 366345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146738 with bad export cookie 4748622897271652858 [11262.551457] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.556140] LustreError: 366345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11262.984886] Lustre: server umount lustre-MDT0001 complete [11273.399118] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11277.156947] Key type lgssc unregistered [11277.446536] LNet: 369898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11277.452751] LNetError: 369898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.502770] LNet: Removed LNI 192.168.202.110@tcp [11279.508363] Key type .llcrypt unregistered [11279.510656] Key type ._llcrypt unregistered [11299.229894] Key type ._llcrypt registered [11299.230950] Key type .llcrypt registered [11299.333272] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11300.666992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11300.690982] alg: No test for adler32 (adler32-zlib) [11301.734232] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [11301.940165] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11303.607181] Key type lgssc registered [11304.733138] Lustre: Echo OBD driver; http://www.lustre.org/ [11311.285961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11311.298704] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.303444] LustreError: 371318:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11311.314801] LustreError: 371318:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11311.320723] LustreError: 371318:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11311.325805] LustreError: 371318:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11311.331840] LustreError: 371318:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11318.092440] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11318.099382] LustreError: 371716:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11318.113380] LustreError: 371716:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11318.118376] LustreError: 371716:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11318.122933] LustreError: 371716:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11318.128958] LustreError: 371716:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11335.883943] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11338.598558] Key type lgssc unregistered [11338.815982] LNet: 372723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11338.820802] LNetError: 372723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11338.836429] LNet: Removed LNI 192.168.202.110@tcp [11339.580533] Key type .llcrypt unregistered [11339.584055] Key type ._llcrypt unregistered [11360.077136] Key type ._llcrypt registered [11360.083576] Key type .llcrypt registered [11360.193403] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [11374.798800] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11375.719750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11375.816826] alg: No test for adler32 (adler32-zlib) [11376.844506] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [11377.050867] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11378.760678] Key type lgssc registered [11379.946864] Lustre: Echo OBD driver; http://www.lustre.org/ [11410.591749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11410.635681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.924526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11411.958626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11412.055928] Lustre: lustre-MDT0000: new disk, initializing [11412.112430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11412.120675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11416.157704] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11427.755500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.950964] Lustre: 375888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11427.988579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11427.993489] Lustre: Skipped 1 previous similar message [11428.073806] Lustre: lustre-MDT0001: new disk, initializing [11428.169505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11428.216603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11428.225432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11433.681753] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11442.579114] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11448.122752] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11453.945703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11454.212384] Lustre: lustre-OST0000: new disk, initializing [11454.217274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11454.224819] Lustre: 377857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11454.306358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11459.741212] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11463.848685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11463.878207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11464.068708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11470.121868] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11475.646604] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11475.870267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11481.055149] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11481.229460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11484.650226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11484.651351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11484.656807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11484.673286] Lustre: Skipped 1 previous similar message [11484.698255] LustreError: Skipped 1 previous similar message [11489.251045] Lustre: server umount lustre-OST0000 complete [11494.887356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.899941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11494.910464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11494.934387] Lustre: Skipped 2 previous similar messages [11499.176445] Lustre: server umount lustre-MDT0000 complete [11504.514872] LustreError: 375881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146980 with bad export cookie 4956253312895290901 [11504.521883] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11504.538299] LustreError: 375881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11505.265758] Lustre: server umount lustre-MDT0001 complete [11514.222862] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11515.862873] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:16:30 (1789146990) [11521.750292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11522.229675] LustreError: 380318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.249067] LustreError: 380318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11522.317573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11526.858150] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11527.660354] LustreError: 380319:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.769539] LustreError: 380318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11536.817389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11537.161422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11541.899456] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11550.067917] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11556.788808] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11565.169628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11565.745369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11572.868426] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11579.885078] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11587.329236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11587.506936] Lustre: lustre-OST0001: new disk, initializing [11587.509963] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11587.515125] Lustre: 383155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11587.593612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11593.035928] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11596.868086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11596.873982] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11596.924411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11601.932317] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11607.034475] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11607.044380] Lustre: Skipped 1 previous similar message [11607.052135] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11607.062138] Lustre: Skipped 1 previous similar message [11612.003401] Lustre: server umount lustre-OST0001 complete [11612.128571] LustreError: 382138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.158060] LustreError: 382138:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11617.267681] LustreError: 382138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11617.295612] LustreError: 382138:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11627.489767] LustreError: 382139:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.525200] LustreError: 382139:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11632.612627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11632.626700] Lustre: Skipped 1 previous similar message [11632.639794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11632.651086] Lustre: Skipped 2 previous similar messages [11637.738504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11637.759857] Lustre: Skipped 1 previous similar message [11645.407170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11645.516590] Lustre: server umount lustre-OST0000 complete [11650.016188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11650.024766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.034287] Lustre: Skipped 1 previous similar message [11650.039322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11650.045601] Lustre: Skipped 2 previous similar messages [11656.104590] Lustre: server umount lustre-MDT0000 complete [11658.217059] LustreError: 380314:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11658.257397] LustreError: 380314:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11660.525258] LustreError: 380300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147136 with bad export cookie 4956253312895291783 [11660.529525] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11660.539212] LustreError: 380300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11667.060553] Lustre: server umount lustre-MDT0001 complete [11676.556223] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [11679.448603] Key type lgssc unregistered [11679.751239] LNet: 385148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11679.757337] LNetError: 385148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11679.775973] LNet: Removed LNI 192.168.202.110@tcp [11680.682403] Key type .llcrypt unregistered [11680.684292] Key type ._llcrypt unregistered [11732.585452] Key type ._llcrypt registered [11732.588363] Key type .llcrypt registered [11732.736559] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [11734.118115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11734.195884] alg: No test for adler32 (adler32-zlib) [11735.406655] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [11735.647710] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [11737.392945] Key type lgssc registered [11738.605179] Lustre: Echo OBD driver; http://www.lustre.org/ [11745.107691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11745.118423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.291743] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11746.305281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11746.429774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11750.976963] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11760.370572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.562290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11760.602389] Lustre: 387950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11760.941396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11765.394882] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11771.919035] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11777.340390] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11781.088443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11781.096691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11785.674085] Lustre: server umount lustre-MDT0000 complete [11789.681928] LustreError: 387943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147265 with bad export cookie 6886152024243427299 [11789.691598] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11790.212948] Lustre: server umount lustre-MDT0001 complete [11800.689717] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:21:14 (1789147274) [11807.988890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11808.460728] LustreError: 390458:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.476351] LustreError: 390458:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11808.507921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11812.995197] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11813.857440] LustreError: 390459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.980223] LustreError: 390458:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.960912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.213752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11827.172962] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11833.877772] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11838.864441] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11844.942885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11845.116932] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11845.360501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11851.065340] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11858.279771] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11867.092616] Lustre: Permanently deactivating lustre-OST0000 [11867.112646] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11874.893321] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11876.700025] LustreError: 393445:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11876.708783] LustreError: 393445:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11876.716534] LustreError: 393445:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11879.441987] LustreError: 393493:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11879.454398] LustreError: 393493:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11887.427241] Lustre: server umount lustre-OST0000 complete [11890.426042] LustreError: 393744:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11890.443779] LustreError: 393744:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11894.245719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11894.257855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11894.278934] Lustre: Skipped 1 previous similar message [11894.296232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11894.301261] Lustre: Skipped 1 previous similar message [11895.272886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.386134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11900.391726] Lustre: Skipped 3 previous similar messages [11902.526798] Lustre: server umount lustre-MDT0000 complete [11905.505322] LustreError: 390459:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11905.539677] LustreError: 390459:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11906.063669] LustreError: 390440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147382 with bad export cookie 6886152024243427880 [11906.067353] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11906.074612] LustreError: 390440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11916.473063] Lustre: server umount lustre-MDT0001 complete [11924.232788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.551668] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11931.091608] LustreError: 394901:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11937.147404] Lustre: 395241:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.110@tcp [11939.306810] Lustre: server umount MGS complete [11951.142513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11951.628996] LustreError: 395965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11951.646763] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11951.654247] Lustre: Skipped 1 previous similar message [11951.720671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11955.976586] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11961.830434] LustreError: 395966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.863197] LustreError: 395966:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11964.864679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11965.236051] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11970.955731] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [11978.563898] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11984.153789] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11986.489080] Lustre: Permanently reactivating lustre-OST0000 [11986.512177] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11986.520287] Lustre: Skipped 2 previous similar messages [12015.071183] LustreError: 397754:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12023.938125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12024.337321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12024.339663] Lustre: Skipped 1 previous similar message [12030.625388] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12040.174147] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12049.892062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12049.897769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.908756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12050.932468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12052.656837] Lustre: server umount lustre-OST0000 complete [12061.156768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12061.171683] Lustre: Skipped 1 previous similar message [12061.183146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12061.185843] Lustre: Skipped 2 previous similar messages [12064.928260] Lustre: server umount lustre-MDT0000 complete [12066.274628] LustreError: 395961:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.309252] LustreError: 395961:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12069.370248] LustreError: 395947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147545 with bad export cookie 6886152024243428930 [12069.373959] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12069.385103] LustreError: 395947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12069.888215] Lustre: server umount lustre-MDT0001 complete [12082.974460] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12086.406290] Key type lgssc unregistered [12086.858875] LNet: 400385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12086.867443] LNetError: 400385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12086.890320] LNet: Removed LNI 192.168.202.110@tcp [12087.785217] Key type .llcrypt unregistered [12087.787293] Key type ._llcrypt unregistered [12105.601365] Key type ._llcrypt registered [12105.607684] Key type .llcrypt registered [12105.771921] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [12123.523268] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12125.238306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12125.304803] alg: No test for adler32 (adler32-zlib) [12126.502652] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [12126.759475] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12128.495154] Key type lgssc registered [12129.642104] Lustre: Echo OBD driver; http://www.lustre.org/ [12167.154144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12167.192133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12168.567748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12168.618402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12168.719679] Lustre: lustre-MDT0000: new disk, initializing [12168.801632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12168.824596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12174.093136] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12185.447529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.684275] Lustre: 403544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12185.807972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12185.815239] Lustre: Skipped 1 previous similar message [12185.980237] Lustre: lustre-MDT0001: new disk, initializing [12186.112370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12186.149309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12186.161135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12190.957780] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12201.134369] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12207.297399] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12214.511168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12214.922364] Lustre: lustre-OST0000: new disk, initializing [12214.929443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12214.939664] Lustre: 405510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12215.041806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12222.562073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12222.584851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12222.695982] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12222.717560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12233.248462] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12238.135915] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12238.394812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12243.782671] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12244.006747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12248.554043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12248.563639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.568405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.608724] Lustre: Skipped 1 previous similar message [12253.676413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12253.681641] Lustre: Skipped 2 previous similar messages [12258.788327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12258.803875] Lustre: Skipped 1 previous similar message [12260.319513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12260.545446] Lustre: server umount lustre-OST0000 complete [12268.004782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12268.022182] LustreError: Skipped 1 previous similar message [12268.027213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12268.049299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12269.029117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12274.153193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12274.156914] Lustre: Skipped 2 previous similar messages [12279.263493] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12279.592436] Lustre: server umount lustre-MDT0000 complete [12284.232281] LustreError: 403534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147760 with bad export cookie 14709976010288826979 [12284.236236] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12284.253319] LustreError: 403534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12284.567789] Lustre: server umount lustre-MDT0001 complete [12293.392556] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12294.997429] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:29:29 (1789147769) [12302.048809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.554280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12307.852824] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12307.938172] LustreError: 407974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.971267] LustreError: 407974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12312.034445] LustreError: 407975:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12317.152087] LustreError: 407974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.246354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12318.850202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12324.152699] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12332.706201] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12340.118364] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12347.615825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12348.174221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12357.986752] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12368.419791] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12370.860572] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12377.961900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12388.326448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12388.329268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12388.335051] Lustre: Skipped 1 previous similar message [12388.340842] Lustre: Skipped 2 previous similar messages [12388.347823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12390.914174] Lustre: server umount lustre-OST0000 complete [12398.561386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.578396] Lustre: Skipped 1 previous similar message [12402.557468] Lustre: server umount lustre-MDT0000 complete [12403.680975] LustreError: 407974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12403.698602] LustreError: 407974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12406.943370] LustreError: 407954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147883 with bad export cookie 14709976010288827861 [12406.950907] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12406.956381] LustreError: 407954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12407.423453] Lustre: server umount lustre-MDT0001 complete [12421.295784] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12425.007169] Key type lgssc unregistered [12425.399088] LNet: 411708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12425.410917] LNetError: 411708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12425.425210] LNet: Removed LNI 192.168.202.110@tcp [12427.126584] Key type .llcrypt unregistered [12427.128212] Key type ._llcrypt unregistered [12458.558373] Key type ._llcrypt registered [12458.561789] Key type .llcrypt registered [12458.807581] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12459.199375] Key type .llcrypt unregistered [12459.201323] Key type ._llcrypt unregistered [12480.064227] Key type ._llcrypt registered [12480.066678] Key type .llcrypt registered [12480.232039] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12481.369877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12481.428186] alg: No test for adler32 (adler32-zlib) [12482.774293] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [12483.147355] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12484.879218] Key type lgssc registered [12486.696288] Lustre: Echo OBD driver; http://www.lustre.org/ [12495.177644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12495.197861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12496.798950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12502.881442] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12506.083174] LustreError: 413941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.105206] LustreError: 413941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12511.206037] LustreError: 413940:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.719510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12513.268249] LustreError: 413941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12513.282289] LustreError: 413941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12513.295255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12518.369499] LustreError: 413940:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12518.405593] LustreError: 413940:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12521.471693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12522.961232] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12526.575465] LustreError: 413941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12526.609790] LustreError: 413941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12533.225133] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12536.811392] LustreError: 414685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.847474] LustreError: 414685:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12542.517671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12542.917577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12549.988183] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12562.407135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12562.423141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12562.453780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12563.428611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12563.429326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12563.433193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12568.556273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12568.565577] Lustre: Skipped 1 previous similar message [12573.676492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12573.687715] Lustre: Skipped 1 previous similar message [12574.176627] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12574.390422] Lustre: server umount lustre-OST0000 complete [12578.786424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12578.795197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12578.810018] Lustre: Skipped 1 previous similar message [12578.827717] Lustre: Skipped 1 previous similar message [12584.913491] Lustre: server umount lustre-MDT0000 complete [12589.037236] LustreError: 413940:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12589.068872] LustreError: 413940:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12589.095937] LustreError: 416048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148065 with bad export cookie 3331620399389363768 [12589.111065] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12589.113256] LustreError: 416048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12589.523681] Lustre: server umount lustre-MDT0001 complete [12603.032664] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12606.473990] Key type lgssc unregistered [12606.725311] LNet: 417095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12606.733707] LNetError: 417095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12606.754317] LNet: Removed LNI 192.168.202.110@tcp [12607.785112] Key type .llcrypt unregistered [12607.787330] Key type ._llcrypt unregistered [12633.848476] Key type ._llcrypt registered [12633.850505] Key type .llcrypt registered [12633.944578] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12635.161405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12635.182360] alg: No test for adler32 (adler32-zlib) [12636.411603] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [12636.790777] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12638.591186] Key type lgssc registered [12639.893946] Lustre: Echo OBD driver; http://www.lustre.org/ [12648.145235] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12648.169556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12663.711442] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12668.831671] LustreError: 418488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.110@tcp: failed processing log, type 4: rc = -110 [12695.520867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12702.365891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12709.522199] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12715.476958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12715.678410] LustreError: 417822:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000647aeed1 x1876057771739392/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12716.190572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12716.305792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12720.752569] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12721.638478] LustreError: 418513:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12721.658948] LustreError: 418513:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12726.753395] LustreError: 418514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12729.821170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12729.828278] LustreError: 418513:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12729.846020] LustreError: 418513:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12730.192489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12736.266468] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12743.947805] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12749.232587] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12757.477494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12757.486418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.508607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12759.231175] Lustre: server umount lustre-OST0000 complete [12763.331198] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12763.337157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12763.345720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12763.950372] Lustre: server umount lustre-MDT0000 complete [12766.177406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12766.178530] LustreError: 419530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.225075] LustreError: 419530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12768.199423] LustreError: 418496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148244 with bad export cookie 13543373628466220084 [12768.212116] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12778.971828] Lustre: server umount lustre-MDT0001 complete [12793.868436] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12796.875683] Key type lgssc unregistered [12797.159310] LNet: 422062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12797.171494] LNetError: 422062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12797.196776] LNet: Removed LNI 192.168.202.110@tcp [12798.070696] Key type .llcrypt unregistered [12798.073151] Key type ._llcrypt unregistered [12825.540687] Key type ._llcrypt registered [12825.542639] Key type .llcrypt registered [12825.648809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12826.672843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12826.729902] alg: No test for adler32 (adler32-zlib) [12827.776148] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [12828.024354] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12829.671181] Key type lgssc registered [12830.813742] Lustre: Echo OBD driver; http://www.lustre.org/ [12839.047685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12839.094787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12840.779730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12845.569659] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12850.150175] LustreError: 423480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.169711] LustreError: 423480:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12854.907322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12855.271377] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12855.436963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12860.004698] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12868.234227] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12874.312824] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12880.528600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12880.826744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12884.927124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12886.687429] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12892.955780] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12899.809916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.811890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12899.835110] Lustre: Skipped 2 previous similar messages [12899.850491] Lustre: Skipped 2 previous similar messages [12903.356893] Lustre: server umount lustre-MDT0000 complete [12904.936618] LustreError: 424496:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.970529] LustreError: 424496:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12906.209608] LustreError: 423461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.10@tcp arrived at 1789148382 with bad export cookie 17391749230685853576 [12908.666315] LustreError: 423463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148384 with bad export cookie 17391749230685852687 [12908.678581] LustreError: 423463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12908.690615] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12910.050062] LustreError: 423477:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12910.052409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12910.093193] LustreError: 423477:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12910.118253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12910.130909] Lustre: Skipped 1 previous similar message [12913.056309] LustreError: 423480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12913.065075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12913.090802] LustreError: 423480:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12914.986238] Lustre: server umount lustre-OST0000 complete [12918.179592] LustreError: 423481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12918.234645] LustreError: 423481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12924.450238] LustreError: 424209:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004cc26a87 x1876057972045696/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12924.793789] Lustre: server umount lustre-MDT0001 complete [12936.633302] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [12939.531505] Key type lgssc unregistered [12939.821097] LNet: 427167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12939.835954] LNetError: 427167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12939.860515] LNet: Removed LNI 192.168.202.110@tcp [12940.786967] Key type .llcrypt unregistered [12940.791474] Key type ._llcrypt unregistered [12964.197243] Key type ._llcrypt registered [12964.200209] Key type .llcrypt registered [12964.346041] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [12965.309684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12965.425519] alg: No test for adler32 (adler32-zlib) [12966.642565] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [12966.868597] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [12968.519328] Key type lgssc registered [12969.788526] Lustre: Echo OBD driver; http://www.lustre.org/ [12979.989338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12980.019474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12981.923117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12987.196756] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [12992.491599] LustreError: 428583:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12992.519979] LustreError: 428583:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12995.238957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.682235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12999.968380] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13008.302598] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13014.301644] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13021.319059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13021.724511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13026.862147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13028.509710] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13036.334954] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13041.635068] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13041.643426] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.668199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13044.310805] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13045.220069] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13047.632508] Lustre: server umount lustre-MDT0001 complete [13049.436474] LustreError: 429824:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.458642] LustreError: 428583:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.460143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13055.499712] LustreError: 428583:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13055.531629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13055.537455] Lustre: Skipped 2 previous similar messages [13060.133862] Lustre: server umount lustre-OST0000 complete [13060.583764] LustreError: 429311:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13065.530396] Lustre: server umount lustre-MDT0000 complete [13080.395368] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13083.524836] Key type lgssc unregistered [13083.780777] LNet: 432273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13083.787917] LNetError: 432273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13083.798932] LNet: Removed LNI 192.168.202.110@tcp [13084.793941] Key type .llcrypt unregistered [13084.799192] Key type ._llcrypt unregistered [13111.300601] Key type ._llcrypt registered [13111.305140] Key type .llcrypt registered [13111.409965] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [13128.564232] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13130.487875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13130.792776] alg: No test for adler32 (adler32-zlib) [13132.225694] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [13132.805148] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13134.567962] Key type lgssc registered [13136.035077] Lustre: Echo OBD driver; http://www.lustre.org/ [13182.476883] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13198.530452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13198.577228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13200.027232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13200.072630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13200.243313] Lustre: lustre-MDT0000: new disk, initializing [13200.457748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13200.501766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13206.902325] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13226.409412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13226.753959] Lustre: lustre-OST0000: new disk, initializing [13226.759918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13226.769424] Lustre: Skipped 1 previous similar message [13226.778473] Lustre: 437308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13226.910473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13233.775484] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13236.951075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13236.971790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13237.028356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13244.022577] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13260.557893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13260.685457] Lustre: 436289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13260.756687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13260.843288] Lustre: lustre-MDT0001: new disk, initializing [13260.938556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13260.999741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13261.017152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13265.689979] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13275.822828] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13276.213245] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13281.431258] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13281.682523] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13288.543747] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.10@tcp (stopping) [13290.977393] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13290.983051] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13290.988791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13291.001410] Lustre: Skipped 1 previous similar message [13293.872400] Lustre: server umount lustre-MDT0001 complete [13298.780669] LustreError: 436296:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.804813] LustreError: 436296:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13301.220768] LustreError: 436300:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13301.232332] LustreError: 436300:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13306.345864] LustreError: 436301:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13306.370673] LustreError: 436301:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13311.456133] LustreError: 436300:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13311.457063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13311.486845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13311.490678] Lustre: Skipped 1 previous similar message [13316.577906] LustreError: 436301:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13316.581665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13317.604697] Lustre: server umount lustre-OST0000 complete [13327.937382] Lustre: server umount lustre-MDT0000 complete [13343.051347] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13346.350570] Key type lgssc unregistered [13346.779682] LNet: 441060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13346.798806] LNetError: 441060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13346.829407] LNet: Removed LNI 192.168.202.110@tcp [13347.882987] Key type .llcrypt unregistered [13347.884675] Key type ._llcrypt unregistered [13388.511519] Key type ._llcrypt registered [13388.514370] Key type .llcrypt registered [13388.625354] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [13388.973421] Key type .llcrypt unregistered [13388.986318] Key type ._llcrypt unregistered [13406.763215] Key type ._llcrypt registered [13406.765975] Key type .llcrypt registered [13406.871271] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [13408.093684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13408.303178] alg: No test for adler32 (adler32-zlib) [13409.372343] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [13409.648110] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [13411.351164] Key type lgssc registered [13412.399022] Lustre: Echo OBD driver; http://www.lustre.org/ [13441.422648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13441.492244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13443.019481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13443.069341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13443.175520] Lustre: lustre-MDT0000: new disk, initializing [13443.288785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13443.309323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13449.596875] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13463.007332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13463.308569] Lustre: 444445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13463.571576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13463.579302] Lustre: Skipped 1 previous similar message [13463.813793] Lustre: lustre-MDT0001: new disk, initializing [13464.176980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13464.265325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13464.288599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13470.354639] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13481.025732] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13488.475791] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13495.626193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13496.123269] Lustre: lustre-OST0000: new disk, initializing [13496.133136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13496.140991] Lustre: 446413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13496.290679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13503.578755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13503.594098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13503.733462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13503.805467] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13514.678716] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13524.453905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13524.455050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13524.467733] Lustre: Skipped 2 previous similar messages [13524.489916] Lustre: Skipped 2 previous similar messages [13528.363158] Lustre: server umount lustre-MDT0000 complete [13532.883850] LustreError: 444436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149009 with bad export cookie 11806007593902926576 [13532.894488] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13532.903581] LustreError: 444436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13533.253969] Lustre: server umount lustre-MDT0001 complete [13548.195863] Lustre: server umount lustre-OST0000 complete [13560.434440] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:50:34 (1789149034) [13569.185093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13569.736418] LustreError: 448442:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13569.761882] LustreError: 448442:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13569.862287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13575.139065] LustreError: 448443:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.319933] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13580.256968] LustreError: 448442:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.014583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.614440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13589.569498] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13596.803462] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13602.810868] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13622.315225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13622.488188] Lustre: Found index 0 for lustre-OST0000, updating log [13622.685862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13631.055029] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [13638.080969] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13641.162192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13649.379265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13649.387286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13649.398843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13650.473242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13650.486895] Lustre: Skipped 1 previous similar message [13654.898120] Lustre: server umount lustre-OST0000 complete [13671.905111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13671.910206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13671.927182] Lustre: Skipped 1 previous similar message [13671.929568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13671.947793] Lustre: Skipped 2 previous similar messages [13676.066373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13676.069379] Lustre: Skipped 1 previous similar message [13677.787498] Lustre: server umount lustre-MDT0000 complete [13681.917833] LustreError: 448424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149158 with bad export cookie 11806007593902927969 [13681.921353] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13681.928907] LustreError: 448424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13682.258596] Lustre: server umount lustre-MDT0001 complete [13685.151197] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [13692.466763] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:52:46 (1789149166) [14312.615534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14313.000484] LustreError: 454867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14313.079862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14317.910261] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14318.571156] LustreError: 454868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14323.682627] LustreError: 454867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14325.887531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14326.304817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14330.705729] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14338.880119] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14344.615522] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14364.066653] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14365.747903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14366.006645] Lustre: Found index 0 for lustre-OST0000, updating log [14366.365371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14372.660442] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14380.997801] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14383.094662] Lustre: server umount lustre-OST0000 complete [14387.682181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14387.697494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14387.710684] Lustre: Skipped 1 previous similar message [14387.723928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14387.732034] Lustre: Skipped 2 previous similar messages [14388.771691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14388.804553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14390.885219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14390.894826] Lustre: Skipped 4 previous similar messages [14393.953846] LustreError: 454867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14393.975488] LustreError: 454867:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14394.109094] Lustre: server umount lustre-MDT0000 complete [14398.056236] LustreError: 454864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14398.078320] LustreError: 454864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [14399.644762] LustreError: 455603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149875 with bad export cookie 11806007593902929054 [14399.651558] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14399.659457] LustreError: 455603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14400.232173] Lustre: server umount lustre-MDT0001 complete [14424.543223] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp [14440.188888] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14453.191121] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14497.304795] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:06:11 (1789149971) [14515.166636] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14518.215313] Key type lgssc unregistered [14518.569763] LNet: 461495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14518.577832] LNetError: 461495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14628.202530] LNet: Removed LNI 192.168.202.110@tcp [14629.077367] Key type .llcrypt unregistered [14629.081724] Key type ._llcrypt unregistered [14644.649969] Key type ._llcrypt registered [14644.652206] Key type .llcrypt registered [14644.753344] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14645.710311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14645.845697] alg: No test for adler32 (adler32-zlib) [14646.953560] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [14647.192488] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [14648.879198] Key type lgssc registered [14650.202946] Lustre: Echo OBD driver; http://www.lustre.org/ [14654.636874] LNet: Added LNI 192.168.202.110@tcp100 [8/256/0/180] [14655.429767] LNet: Added LNI 192.168.202.110@tcp101 [8/256/0/180] [14656.381633] LNet: Added LNI 192.168.202.110@tcp102 [8/256/0/180] [14657.321772] LNet: Added LNI 192.168.202.110@tcp103 [8/256/0/180] [14658.151702] LNet: Added LNI 192.168.202.110@tcp104 [8/256/0/180] [14659.029133] LNet: Added LNI 192.168.202.110@tcp105 [8/256/0/180] [14659.811842] LNet: Added LNI 192.168.202.110@tcp106 [8/256/0/180] [14660.626434] LNet: Added LNI 192.168.202.110@tcp107 [8/256/0/180] [14661.513609] LNet: Added LNI 192.168.202.110@tcp108 [8/256/0/180] [14662.453994] LNet: Added LNI 192.168.202.110@tcp109 [8/256/0/180] [14663.421254] LNet: Added LNI 192.168.202.110@tcp110 [8/256/0/180] [14664.317646] LNet: Added LNI 192.168.202.110@tcp111 [8/256/0/180] [14665.360822] LNet: Added LNI 192.168.202.110@tcp112 [8/256/0/180] [14666.323489] LNet: Added LNI 192.168.202.110@tcp113 [8/256/0/180] [14667.122834] LNet: Added LNI 192.168.202.110@tcp114 [8/256/0/180] [14667.982667] LNet: Added LNI 192.168.202.110@tcp115 [8/256/0/180] [14668.924820] LNet: Added LNI 192.168.202.110@tcp116 [8/256/0/180] [14669.766548] LNet: Added LNI 192.168.202.110@tcp117 [8/256/0/180] [14670.796918] LNet: Added LNI 192.168.202.110@tcp118 [8/256/0/180] [14671.949750] LNet: Added LNI 192.168.202.110@tcp119 [8/256/0/180] [14672.962522] LNet: Added LNI 192.168.202.110@tcp120 [8/256/0/180] [14673.867281] LNet: Added LNI 192.168.202.110@tcp121 [8/256/0/180] [14674.890996] LNet: Added LNI 192.168.202.110@tcp122 [8/256/0/180] [14675.684445] LNet: Added LNI 192.168.202.110@tcp123 [8/256/0/180] [14676.405480] LNet: Added LNI 192.168.202.110@tcp124 [8/256/0/180] [14677.187490] LNet: Added LNI 192.168.202.110@tcp125 [8/256/0/180] [14678.042730] LNet: Added LNI 192.168.202.110@tcp126 [8/256/0/180] [14678.826586] LNet: Added LNI 192.168.202.110@tcp127 [8/256/0/180] [14679.463969] LNet: Added LNI 192.168.202.110@tcp128 [8/256/0/180] [14680.218687] LNet: Added LNI 192.168.202.110@tcp129 [8/256/0/180] [14680.858436] LNet: Added LNI 192.168.202.110@tcp130 [8/256/0/180] [14681.886971] LNet: Added LNI 192.168.202.110@tcp131 [8/256/0/180] [14682.789409] LNet: Added LNI 192.168.202.110@tcp132 [8/256/0/180] [14683.679518] LNet: Added LNI 192.168.202.110@tcp133 [8/256/0/180] [14684.577783] LNet: Added LNI 192.168.202.110@tcp134 [8/256/0/180] [14685.392344] LNet: Added LNI 192.168.202.110@tcp135 [8/256/0/180] [14691.988364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14692.013424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14693.170345] Lustre: 464526:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14693.251177] LNetError: 464554:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14693.266128] LNetError: Protocol error connecting to 192.168.202.110@tcp112 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14693.282111] LNetError: Skipped 6 previous similar messages [14693.285192] LNet: 5 local NIs in recovery (showing 5): 192.168.202.110@tcp124, 192.168.202.110@tcp125, 192.168.202.110@tcp121, 192.168.202.110@tcp123, 192.168.202.110@tcp129 [14693.389549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14693.422713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14693.526901] Lustre: lustre-MDT0000: new disk, initializing [14693.619873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14693.634827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14694.371623] LNetError: 461982:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14694.394602] LNetError: 461982:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14694.431149] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14694.452483] LNetError: Skipped 5 previous similar messages [14696.416519] LNetError: 464547:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14696.440806] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14699.312326] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14700.527340] LNetError: 461981:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14700.542985] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14708.704897] LNetError: 464559:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14708.711988] LNetError: Protocol error connecting to 192.168.202.110@tcp123 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14711.141245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14711.460646] Lustre: 464543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14711.522366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14711.523911] Lustre: Skipped 1 previous similar message [14711.669535] Lustre: lustre-MDT0001: new disk, initializing [14711.731298] Lustre: 465346:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14711.752947] Lustre: 465346:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14711.899176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14711.942384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14711.956959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14716.511891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14721.575531] Lustre: 465789:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14721.580935] Lustre: 465789:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14724.003720] LNetError: 461983:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14724.034335] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14725.851753] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14731.948485] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14739.661954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14739.930845] Lustre: lustre-OST0000: new disk, initializing [14739.934452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14739.938556] Lustre: 466518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14739.999234] Lustre: 466523:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14740.003591] Lustre: 466523:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14740.044683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14740.385837] LNetError: 461981:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14740.393371] LNetError: 461981:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14740.398605] LNetError: Protocol error connecting to 192.168.202.110@tcp118 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14740.408329] LNetError: Skipped 6 previous similar messages [14747.174393] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [14750.271488] Lustre: 466994:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.110@tcp: can't add NID 192.168.202.110@tcp130: rc = -75 [14750.283859] Lustre: 466994:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14750.355347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14750.373680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14750.551120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14752.799163] LNet: 1 local NIs in recovery (showing 1): 192.168.202.110@tcp123 [14756.653415] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14766.049064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14766.049258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14766.057115] LustreError: Skipped 1 previous similar message [14766.076207] Lustre: Skipped 1 previous similar message [14766.078589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14766.083624] Lustre: Skipped 1 previous similar message [14771.182431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14771.187987] Lustre: Skipped 1 previous similar message [14776.291279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14776.293905] Lustre: Skipped 1 previous similar message [14780.383336] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14780.504499] Lustre: server umount lustre-OST0000 complete [14786.529349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14786.559679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14786.577205] Lustre: Skipped 1 previous similar message [14788.576382] LNetError: 461983:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.110 [14788.596581] LNetError: 461983:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [14788.604248] LNetError: Protocol error connecting to 192.168.202.110@tcp120 at host 192.168.202.110:988: is it running a compatible version of Lustre? [14788.617732] LNetError: Skipped 14 previous similar messages [14791.651661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14791.670762] Lustre: Skipped 1 previous similar message [14799.327508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14799.677967] Lustre: server umount lustre-MDT0000 complete [14803.736616] LustreError: 464534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150279 with bad export cookie 18277682899603120514 [14803.741603] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14803.749779] LustreError: 464534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14804.076040] Lustre: server umount lustre-MDT0001 complete [14817.049835] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14819.868457] Key type lgssc unregistered [14820.369863] LNet: 468439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14820.381899] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14820.398957] LNet: Removed LNI 192.168.202.110@tcp [14821.407167] LNet: Removed LNI 192.168.202.110@tcp100 [14821.409231] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14821.417919] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14821.430228] LNet: Removed LNI 192.168.202.110@tcp101 [14822.495758] LNet: Removed LNI 192.168.202.110@tcp102 [14822.514905] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14822.521558] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14822.528484] LNet: Removed LNI 192.168.202.110@tcp103 [14823.583184] LNet: Removed LNI 192.168.202.110@tcp104 [14824.607170] LNet: Removed LNI 192.168.202.110@tcp105 [14824.614771] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14824.621112] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14824.635896] LNet: Removed LNI 192.168.202.110@tcp106 [14825.695168] LNet: Removed LNI 192.168.202.110@tcp107 [14826.719538] LNet: Removed LNI 192.168.202.110@tcp108 [14827.743746] LNet: Removed LNI 192.168.202.110@tcp109 [14828.831289] LNet: Removed LNI 192.168.202.110@tcp110 [14828.833393] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14828.845824] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14828.853513] LNet: Removed LNI 192.168.202.110@tcp111 [14829.920752] LNet: Removed LNI 192.168.202.110@tcp112 [14831.007224] LNet: Removed LNI 192.168.202.110@tcp113 [14832.031320] LNet: Removed LNI 192.168.202.110@tcp114 [14833.055319] LNet: Removed LNI 192.168.202.110@tcp115 [14834.079232] LNet: Removed LNI 192.168.202.110@tcp116 [14835.103630] LNet: Removed LNI 192.168.202.110@tcp117 [14836.127157] LNet: Removed LNI 192.168.202.110@tcp118 [14837.151169] LNet: Removed LNI 192.168.202.110@tcp119 [14837.153149] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14837.164627] LNetError: 468439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14837.193266] LNet: Removed LNI 192.168.202.110@tcp120 [14838.249461] LNet: Removed LNI 192.168.202.110@tcp121 [14839.264774] LNet: Removed LNI 192.168.202.110@tcp122 [14839.266934] LNet: Removed LNI 192.168.202.110@tcp123 [14840.351171] LNet: Removed LNI 192.168.202.110@tcp124 [14841.378614] LNet: Removed LNI 192.168.202.110@tcp125 [14842.399172] LNet: Removed LNI 192.168.202.110@tcp126 [14843.423426] LNet: Removed LNI 192.168.202.110@tcp127 [14844.447187] LNet: Removed LNI 192.168.202.110@tcp128 [14845.471171] LNet: Removed LNI 192.168.202.110@tcp129 [14845.479858] LNet: Removed LNI 192.168.202.110@tcp130 [14845.485702] LNet: Removed LNI 192.168.202.110@tcp131 [14845.494959] LNet: Removed LNI 192.168.202.110@tcp132 [14845.498912] LNet: Removed LNI 192.168.202.110@tcp133 [14845.503804] LNet: Removed LNI 192.168.202.110@tcp134 [14845.529268] LNet: Removed LNI 192.168.202.110@tcp135 [14847.305905] Key type .llcrypt unregistered [14847.309297] Key type ._llcrypt unregistered [14874.871638] Key type ._llcrypt registered [14874.873374] Key type .llcrypt registered [14875.012929] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [14892.475744] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14894.300523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14894.380195] alg: No test for adler32 (adler32-zlib) [14895.512198] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [14895.742515] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [14897.407191] Key type lgssc registered [14899.152025] Lustre: Echo OBD driver; http://www.lustre.org/ [14943.117818] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:13:37 (1789150417) [14963.810617] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [14968.016449] Key type lgssc unregistered [14968.431659] LNet: 472533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14968.437096] LNetError: 472533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14968.452680] LNet: Removed LNI 192.168.202.110@tcp [14969.077183] Key type .llcrypt unregistered [14969.078746] Key type ._llcrypt unregistered [14987.635111] Key type ._llcrypt registered [14987.636993] Key type .llcrypt registered [14987.739180] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [14988.955963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14989.010546] alg: No test for adler32 (adler32-zlib) [14990.172257] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [14990.383378] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [14992.087450] Key type lgssc registered [14993.348518] Lustre: Echo OBD driver; http://www.lustre.org/ [15004.180881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15004.207206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15005.544458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15005.570298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15005.642552] Lustre: lustre-MDT0000: new disk, initializing [15005.724411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15005.732726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15011.861320] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15026.667275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15026.838710] Lustre: 473746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15026.932274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15026.946695] Lustre: Skipped 1 previous similar message [15027.155155] Lustre: lustre-MDT0001: new disk, initializing [15027.326547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15027.369382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15027.376564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15031.645713] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15042.720560] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15047.969967] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15055.010441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15055.371387] Lustre: lustre-OST0000: new disk, initializing [15055.379911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15055.394429] Lustre: 475718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15055.516370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15061.876432] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15062.204578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15062.219844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15062.390818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15072.797232] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15078.163374] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing lnet_if_list [15083.557093] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15083.569794] LNet: Added LNI 192.168.202.110@tcp10 [8/256/0/180] [15088.855670] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15088.870326] LNet: Added LNI 192.168.202.110@tcp11 [8/256/0/180] [15094.698507] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15094.732728] LNet: Added LNI 192.168.202.110@tcp12 [8/256/0/180] [15100.651367] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15100.716707] LNet: Added LNI 192.168.202.110@tcp13 [8/256/0/180] [15106.805936] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15106.853333] LNet: Added LNI 192.168.202.110@tcp14 [8/256/0/180] [15112.680916] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15112.704585] LNet: Added LNI 192.168.202.110@tcp15 [8/256/0/180] [15119.067176] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15119.123590] LNet: Added LNI 192.168.202.110@tcp16 [8/256/0/180] [15126.212439] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15126.259111] LNet: Added LNI 192.168.202.110@tcp17 [8/256/0/180] [15132.402752] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15132.457687] LNet: Added LNI 192.168.202.110@tcp18 [8/256/0/180] [15137.918966] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15137.940422] LNet: Added LNI 192.168.202.110@tcp19 [8/256/0/180] [15144.803914] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15144.838891] LNet: Added LNI 192.168.202.110@tcp20 [8/256/0/180] [15150.147465] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15150.187811] LNet: Added LNI 192.168.202.110@tcp21 [8/256/0/180] [15156.600685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15156.635317] LNet: Added LNI 192.168.202.110@tcp22 [8/256/0/180] [15163.410968] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15163.432832] LNet: Added LNI 192.168.202.110@tcp23 [8/256/0/180] [15169.125762] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15169.150472] LNet: Added LNI 192.168.202.110@tcp24 [8/256/0/180] [15175.363423] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15175.392751] LNet: Added LNI 192.168.202.110@tcp25 [8/256/0/180] [15181.102641] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15181.125858] LNet: Added LNI 192.168.202.110@tcp26 [8/256/0/180] [15186.915761] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15186.934306] LNet: Added LNI 192.168.202.110@tcp27 [8/256/0/180] [15194.780160] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15194.819848] LNet: Added LNI 192.168.202.110@tcp28 [8/256/0/180] [15203.208565] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15203.243779] LNet: Added LNI 192.168.202.110@tcp29 [8/256/0/180] [15211.639738] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15211.735261] LNet: Added LNI 192.168.202.110@tcp30 [8/256/0/180] [15220.797857] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15220.866717] LNet: Added LNI 192.168.202.110@tcp31 [8/256/0/180] [15228.675159] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15228.708396] LNet: Added LNI 192.168.202.110@tcp32 [8/256/0/180] [15235.136522] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15235.175610] LNet: Added LNI 192.168.202.110@tcp33 [8/256/0/180] [15244.074239] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15244.117632] LNet: Added LNI 192.168.202.110@tcp34 [8/256/0/180] [15252.956686] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15253.038518] LNet: Added LNI 192.168.202.110@tcp35 [8/256/0/180] [15263.674731] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15263.720759] LNet: Added LNI 192.168.202.110@tcp36 [8/256/0/180] [15273.084619] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15273.110679] LNet: Added LNI 192.168.202.110@tcp37 [8/256/0/180] [15280.518747] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15280.585039] LNet: Added LNI 192.168.202.110@tcp38 [8/256/0/180] [15288.441250] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15288.475796] LNet: Added LNI 192.168.202.110@tcp39 [8/256/0/180] [15296.822722] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15296.862344] LNet: Added LNI 192.168.202.110@tcp40 [8/256/0/180] [15304.802746] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15304.842578] LNet: Added LNI 192.168.202.110@tcp41 [8/256/0/180] [15313.000911] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15313.051871] LNet: Added LNI 192.168.202.110@tcp42 [8/256/0/180] [15321.163717] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15321.210464] LNet: Added LNI 192.168.202.110@tcp43 [8/256/0/180] [15329.883789] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15329.915422] LNet: Added LNI 192.168.202.110@tcp44 [8/256/0/180] [15337.016819] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15337.056938] LNet: Added LNI 192.168.202.110@tcp45 [8/256/0/180] [15345.350631] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15345.394804] LNet: Added LNI 192.168.202.110@tcp46 [8/256/0/180] [15353.622537] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15353.665442] LNet: Added LNI 192.168.202.110@tcp47 [8/256/0/180] [15363.870423] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15363.896819] LNet: Added LNI 192.168.202.110@tcp48 [8/256/0/180] [15372.249251] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15372.268874] LNet: Added LNI 192.168.202.110@tcp49 [8/256/0/180] [15382.237856] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15382.259893] LNet: Added LNI 192.168.202.110@tcp50 [8/256/0/180] [15400.424010] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15400.438270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15400.439447] Lustre: Skipped 1 previous similar message [15400.454469] Lustre: Skipped 2 previous similar messages [15405.540569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15405.562299] Lustre: Skipped 2 previous similar messages [15406.991479] Lustre: server umount lustre-MDT0000 complete [15411.224721] LustreError: 473740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150887 with bad export cookie 6399064426954705156 [15411.232051] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15411.239205] LustreError: 473740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15411.520610] Lustre: server umount lustre-MDT0001 complete [15430.522642] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15432.172087] Lustre: 473021:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150892/real 1789150892] req@0000000014523558 x1876060239220096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15432.215962] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15432.229320] Lustre: Skipped 1 previous similar message [15433.185714] Lustre: 473020:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150893/real 1789150893] req@00000000549b0cc2 x1876060239220224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15437.345258] Lustre: 473020:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150897/real 1789150897] req@000000001cebeac2 x1876060239220352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15439.733141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15442.399354] Lustre: 473021:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150902/real 1789150902] req@00000000549f95df x1876060239220608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15447.521099] Lustre: 473022:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789150907/real 1789150907] req@0000000076c7d9c8 x1876060239220736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789150923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15456.099189] LustreError: 475712:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15456.117959] LustreError: 475712:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [15456.172302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15457.185395] LustreError: 475713:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15457.201590] LustreError: 475713:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15460.412976] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15462.370759] LustreError: 499964:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15462.401075] LustreError: 499964:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15466.479513] LustreError: 475712:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15466.480928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15466.495154] LustreError: 475712:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15470.015065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15470.451373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15475.110584] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15475.701647] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15483.103610] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15489.351164] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15511.540210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15511.547173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15511.552350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15511.560318] Lustre: Skipped 1 previous similar message [15515.947985] Lustre: server umount lustre-OST0000 complete [15521.772492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15521.772734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15521.781115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15521.804444] Lustre: Skipped 1 previous similar message [15521.816287] Lustre: Skipped 2 previous similar messages [15525.670709] Lustre: server umount lustre-MDT0000 complete [15526.880242] LustreError: 499969:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15526.897032] LustreError: 499969:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15529.269480] LustreError: 473738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151005 with bad export cookie 6399064426954707774 [15529.273461] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15529.289676] LustreError: 473738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15529.853375] Lustre: server umount lustre-MDT0001 complete [15541.208107] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15544.660437] Key type lgssc unregistered [15545.135973] LNet: 502764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15545.146976] LNetError: 502764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15546.207184] LNet: Removed LNI 192.168.202.110@tcp [15546.214371] LNetError: 502764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15546.229903] LNet: Removed LNI 192.168.202.110@tcp10 [15546.236438] LNet: Removed LNI 192.168.202.110@tcp11 [15546.256714] LNet: Removed LNI 192.168.202.110@tcp12 [15546.270515] LNet: Removed LNI 192.168.202.110@tcp13 [15546.279949] LNet: Removed LNI 192.168.202.110@tcp14 [15546.305794] LNet: Removed LNI 192.168.202.110@tcp15 [15546.312267] LNet: Removed LNI 192.168.202.110@tcp16 [15546.324802] LNet: Removed LNI 192.168.202.110@tcp17 [15546.334406] LNet: Removed LNI 192.168.202.110@tcp18 [15546.347040] LNet: Removed LNI 192.168.202.110@tcp19 [15546.354915] LNet: Removed LNI 192.168.202.110@tcp20 [15546.358732] LNet: Removed LNI 192.168.202.110@tcp21 [15546.372790] LNet: Removed LNI 192.168.202.110@tcp22 [15546.382250] LNet: Removed LNI 192.168.202.110@tcp23 [15546.387646] LNet: Removed LNI 192.168.202.110@tcp24 [15546.389570] LNet: Removed LNI 192.168.202.110@tcp25 [15546.395801] LNet: Removed LNI 192.168.202.110@tcp26 [15546.400923] LNet: Removed LNI 192.168.202.110@tcp27 [15546.405000] LNet: Removed LNI 192.168.202.110@tcp28 [15546.409918] LNet: Removed LNI 192.168.202.110@tcp29 [15546.411932] LNet: Removed LNI 192.168.202.110@tcp30 [15546.415763] LNet: Removed LNI 192.168.202.110@tcp31 [15546.418851] LNet: Removed LNI 192.168.202.110@tcp32 [15546.426418] LNet: Removed LNI 192.168.202.110@tcp33 [15546.432331] LNet: Removed LNI 192.168.202.110@tcp34 [15546.436290] LNet: Removed LNI 192.168.202.110@tcp35 [15546.444818] LNet: Removed LNI 192.168.202.110@tcp36 [15546.451655] LNet: Removed LNI 192.168.202.110@tcp37 [15546.460892] LNet: Removed LNI 192.168.202.110@tcp38 [15546.464690] LNet: Removed LNI 192.168.202.110@tcp39 [15546.467477] LNet: Removed LNI 192.168.202.110@tcp40 [15546.469488] LNet: Removed LNI 192.168.202.110@tcp41 [15546.480781] LNet: Removed LNI 192.168.202.110@tcp42 [15546.485042] LNet: Removed LNI 192.168.202.110@tcp43 [15546.492199] LNet: Removed LNI 192.168.202.110@tcp44 [15546.494074] LNet: Removed LNI 192.168.202.110@tcp45 [15546.503898] LNet: Removed LNI 192.168.202.110@tcp46 [15546.508395] LNet: Removed LNI 192.168.202.110@tcp47 [15546.513936] LNet: Removed LNI 192.168.202.110@tcp48 [15546.520827] LNet: Removed LNI 192.168.202.110@tcp49 [15546.534932] LNet: Removed LNI 192.168.202.110@tcp50 [15547.584534] Key type .llcrypt unregistered [15547.588608] Key type ._llcrypt unregistered [15575.090122] Key type ._llcrypt registered [15575.099106] Key type .llcrypt registered [15575.322558] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [15592.184812] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15593.184503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15593.238635] alg: No test for adler32 (adler32-zlib) [15594.309934] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [15594.561116] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15596.295598] Key type lgssc registered [15598.047856] Lustre: Echo OBD driver; http://www.lustre.org/ [15643.819384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15643.872939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15645.294703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15645.325804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15645.452127] Lustre: lustre-MDT0000: new disk, initializing [15645.561489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15645.588515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15650.768328] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15661.597231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15661.792351] Lustre: 505963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15661.875253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15661.880057] Lustre: Skipped 1 previous similar message [15662.018292] Lustre: lustre-MDT0001: new disk, initializing [15662.158018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15662.196638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15662.213374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15668.437133] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15679.042090] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15684.760751] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15693.186514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15693.422915] Lustre: *** cfs_fail_loc=90f, val=0*** [15693.532204] Lustre: lustre-OST0000: new disk, initializing [15693.544439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15693.560942] Lustre: 507979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15693.698944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15700.792271] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15702.175936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15702.189765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15702.328170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15711.878087] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15722.990172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15722.998088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15723.002063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15723.010467] Lustre: Skipped 1 previous similar message [15725.521564] Lustre: server umount lustre-OST0000 complete [15733.218399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15733.225830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15733.242610] Lustre: Skipped 1 previous similar message [15733.257889] Lustre: Skipped 1 previous similar message [15735.790831] Lustre: server umount lustre-MDT0000 complete [15738.337365] LustreError: 506248:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15738.373068] LustreError: 506248:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [15739.814382] LustreError: 507989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151216 with bad export cookie 14600028562824844608 [15739.825643] LustreError: 507989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15739.828034] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15740.201896] Lustre: server umount lustre-MDT0001 complete [15751.664749] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15755.117796] Key type lgssc unregistered [15755.455924] LNet: 509890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15755.471389] LNetError: 509890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15755.494823] LNet: Removed LNI 192.168.202.110@tcp [15756.723308] Key type .llcrypt unregistered [15756.727187] Key type ._llcrypt unregistered [15778.187438] Key type ._llcrypt registered [15778.191112] Key type .llcrypt registered [15778.298220] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [15779.395651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15779.517340] alg: No test for adler32 (adler32-zlib) [15780.730239] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [15781.031060] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [15782.759715] Key type lgssc registered [15784.271239] Lustre: Echo OBD driver; http://www.lustre.org/ [15793.483788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15793.511357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15794.993932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15800.507170] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15805.424550] LustreError: 511158:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15805.460506] LustreError: 511158:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15809.370835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15809.818232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15814.397585] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15822.019273] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15827.505192] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15835.248455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15835.782248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15842.932290] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15850.488499] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15864.155382] loop0: detected capacity change from 0 to 204800000 [15864.353074] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15864.375965] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15864.396355] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15870.683662] loop0: detected capacity change from 0 to 204800000 [15870.908704] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15870.933626] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15870.947648] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15876.190909] loop0: detected capacity change from 0 to 204800000 [15881.567525] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15881.754758] Lustre: fs15246-OST0000: new disk, initializing [15881.760620] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15881.770546] Lustre: 514750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15881.864914] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15888.744249] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15898.169317] loop1: detected capacity change from 0 to 204800000 [15903.362887] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15903.501400] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15903.576472] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15903.663076] Lustre: fs15246-MDT0000: new disk, initializing [15903.917416] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15903.948055] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15904.072886] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15905.121119] Lustre: 515874:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000355767ec x1876061068507776/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789151397 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15907.995209] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [15914.209930] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15914.233660] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15914.266495] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15926.756719] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15926.770487] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15931.882924] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15933.129860] Lustre: server umount fs15246-MDT0000 complete [15937.008462] LustreError: 511155:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15940.068136] LustreError: 511155:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15941.938770] Lustre: server umount fs15246-OST0000 complete [15955.428689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15955.436382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15955.464149] Lustre: Skipped 1 previous similar message [15955.496103] Lustre: Skipped 1 previous similar message [15960.553359] LustreError: 512982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15961.062287] Lustre: server umount lustre-OST0000 complete [15968.736561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15968.753170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15968.789909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15970.785341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15972.695486] Lustre: server umount lustre-MDT0000 complete [15975.919203] LustreError: 511159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15975.956151] LustreError: 511159:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15977.211821] LustreError: 511140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151453 with bad export cookie 12324312094154616592 [15977.232722] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15977.736555] Lustre: server umount lustre-MDT0001 complete [15993.517989] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [15998.648935] Key type lgssc unregistered [15999.393602] LNet: 518976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15999.406351] LNetError: 518976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15999.442449] LNet: Removed LNI 192.168.202.110@tcp [16000.691212] Key type .llcrypt unregistered [16000.693041] Key type ._llcrypt unregistered [16020.401592] Key type ._llcrypt registered [16020.405895] Key type .llcrypt registered [16020.493343] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16021.316482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16021.418806] alg: No test for adler32 (adler32-zlib) [16022.499186] Lustre: Lustre: Build Version: 2.17.57_127_ged01a19 [16022.807985] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [16024.535616] Key type lgssc registered [16025.609413] Lustre: Echo OBD driver; http://www.lustre.org/ [16063.416790] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16073.190562] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16108.193766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16108.215786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16109.536699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16109.579203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16109.691938] Lustre: lustre-MDT0000: new disk, initializing [16109.819754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16109.837590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16114.395259] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16124.995523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16125.150969] Lustre: 523446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16125.201663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16125.204465] Lustre: Skipped 1 previous similar message [16125.289757] Lustre: lustre-MDT0001: new disk, initializing [16125.397104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16125.435349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16125.453367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16129.624852] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16139.354390] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16145.357776] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16152.804116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16153.052754] Lustre: lustre-OST0000: new disk, initializing [16153.057631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16153.067990] Lustre: 525414:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16153.188543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16159.373593] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug -1 all [16161.969472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16161.987284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16162.130046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16169.574843] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16174.268598] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16174.532037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16180.256068] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16180.523101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16182.753816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16182.758289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16182.765529] LustreError: Skipped 1 previous similar message [16182.769970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16182.774848] Lustre: Skipped 1 previous similar message [16182.783338] Lustre: Skipped 1 previous similar message [16187.875959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16187.886234] Lustre: Skipped 1 previous similar message [16188.882020] Lustre: server umount lustre-OST0000 complete [16191.968530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16191.973556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16191.985981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16193.009584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16198.125175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16198.134145] Lustre: Skipped 3 previous similar messages [16203.232451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16203.239890] Lustre: Skipped 1 previous similar message [16206.303189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16206.593859] Lustre: server umount lustre-MDT0000 complete [16210.298480] LustreError: 523439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151686 with bad export cookie 6366594889545752583 [16210.301726] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16210.312805] LustreError: 523439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16210.598671] Lustre: server umount lustre-MDT0001 complete [16234.651089] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [16245.163797] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [16274.036620] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15972 sec ========= 14:35:48 (1789151748) [16275.563190] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:35:49 (1789151749) === [16278.476814] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:35:52 (1789151752) === [16316.978847] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [16319.675757] Key type lgssc unregistered [16319.973819] LNet: 532064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16319.979687] LNetError: 532064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16320.000134] LNet: Removed LNI 192.168.202.110@tcp [16320.922365] Key type .llcrypt unregistered [16320.927174] Key type ._llcrypt unregistered