[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443749350 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002236] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006322] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009004] pid_max: default: 32768 minimum: 301 [ 0.010086] LSM: Security Framework initializing [ 0.011025] Yama: becoming mindful. [ 0.011809] SELinux: Initializing. [ 0.012038] *** VALIDATE selinux *** [ 0.019228] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023039] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024086] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025065] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026058] *** VALIDATE tmpfs *** [ 0.028135] *** VALIDATE proc *** [ 0.029063] *** VALIDATE cgroup *** [ 0.029858] *** VALIDATE cgroup2 *** [ 0.031093] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032108] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034017] Spectre V2 : User space: Vulnerable [ 0.035003] Speculative Store Bypass: Vulnerable [ 0.038243] debug: unmapping init [mem 0xffffffffac059000-0xffffffffac060fff] [ 0.040151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041418] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042009] ... version: 2 [ 0.042881] ... bit width: 48 [ 0.043006] ... generic registers: 4 [ 0.043875] ... value mask: 0000ffffffffffff [ 0.044006] ... max period: 00007fffffffffff [ 0.045006] ... fixed-purpose events: 3 [ 0.045867] ... event mask: 000000070000000f [ 0.046173] rcu: Hierarchical SRCU implementation. [ 0.048093] smp: Bringing up secondary CPUs ... [ 0.049394] x86: Booting SMP configuration: [ 0.050014] .... node #0, CPUs: #1 #2 #3 [ 0.052525] smp: Brought up 1 node, 4 CPUs [ 0.053892] smpboot: Max logical packages: 1 [ 0.054006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148609] node 0 deferred pages initialised in 93ms [ 0.151454] devtmpfs: initialized [ 0.152153] x86/mm: Memory block size: 128MB [ 0.154306] gcov: version magic: 0x41383552 [ 0.155535] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157167] pinctrl core: initialized pinctrl subsystem [ 0.158099] [ 0.158394] ************************************************************* [ 0.159006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160006] ** ** [ 0.161006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162006] ** ** [ 0.163006] ** This means that this kernel is built to expose internal ** [ 0.164006] ** IOMMU data structures, which may compromise security on ** [ 0.165006] ** your system. ** [ 0.166006] ** ** [ 0.167007] ** If you see this message and you are not debugging the ** [ 0.168006] ** kernel, report this immediately to your vendor! ** [ 0.169006] ** ** [ 0.170006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171006] ************************************************************* [ 0.172506] NET: Registered protocol family 16 [ 0.173321] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176336] cpuidle: using governor menu [ 0.177348] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179436] PCI: Using configuration type 1 for base access [ 0.180092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188125] cryptd: max_cpu_qlen set to 1000 [ 0.191846] ACPI: Added _OSI(Module Device) [ 0.193011] ACPI: Added _OSI(Processor Device) [ 0.195010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202781] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209232] ACPI: Interpreter enabled [ 0.210034] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210989] ACPI: Using IOAPIC for interrupt routing [ 0.212065] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215246] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222775] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.224017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228047] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232989] acpiphp: Slot [2] registered [ 0.234085] acpiphp: Slot [5] registered [ 0.236081] acpiphp: Slot [6] registered [ 0.237083] acpiphp: Slot [7] registered [ 0.238018] acpiphp: Slot [8] registered [ 0.238969] acpiphp: Slot [9] registered [ 0.240062] acpiphp: Slot [10] registered [ 0.241092] acpiphp: Slot [3] registered [ 0.243066] acpiphp: Slot [4] registered [ 0.244034] acpiphp: Slot [11] registered [ 0.245052] acpiphp: Slot [12] registered [ 0.246081] acpiphp: Slot [13] registered [ 0.247048] acpiphp: Slot [14] registered [ 0.248046] acpiphp: Slot [15] registered [ 0.248967] acpiphp: Slot [16] registered [ 0.250071] acpiphp: Slot [17] registered [ 0.251078] acpiphp: Slot [18] registered [ 0.253047] acpiphp: Slot [19] registered [ 0.254000] acpiphp: Slot [20] registered [ 0.254908] acpiphp: Slot [21] registered [ 0.255082] acpiphp: Slot [22] registered [ 0.255976] acpiphp: Slot [23] registered [ 0.257053] acpiphp: Slot [24] registered [ 0.257921] acpiphp: Slot [25] registered [ 0.259054] acpiphp: Slot [26] registered [ 0.260022] acpiphp: Slot [27] registered [ 0.262080] acpiphp: Slot [28] registered [ 0.264068] acpiphp: Slot [29] registered [ 0.265068] acpiphp: Slot [30] registered [ 0.267074] acpiphp: Slot [31] registered [ 0.269045] PCI host bridge to bus 0000:00 [ 0.270010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.271016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.274012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.277026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282141] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284706] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286787] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293008] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.297335] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.300011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.305011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308598] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.311528] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.316724] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.320881] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.329009] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.332007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.335563] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.346007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.359025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.365251] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.378010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.391011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.402301] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.408009] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.412008] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.421010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.429969] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.433008] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.437008] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.445010] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.454178] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.461010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.466010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.475011] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.484750] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.488009] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.492008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.502008] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.509109] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.511221] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.513190] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.515194] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.516118] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.519087] iommu: Default domain type: Passthrough [ 0.522238] SCSI subsystem initialized [ 0.523069] ACPI: bus type USB registered [ 0.523976] usbcore: registered new interface driver usbfs [ 0.525034] usbcore: registered new interface driver hub [ 0.526027] usbcore: registered new device driver usb [ 0.527100] pps_core: LinuxPPS API ver. 1 registered [ 0.529007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.531025] PTP clock support registered [ 0.533044] EDAC MC: Ver: 3.0.0 [ 0.534486] PCI: Using ACPI for IRQ routing [ 0.536606] NetLabel: Initializing [ 0.537005] NetLabel: domain hash size = 128 [ 0.538006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.540038] NetLabel: unlabeled traffic allowed by default [ 0.541073] vgaarb: loaded [ 0.542287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.544006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.551273] clocksource: Switched to clocksource kvm-clock [ 0.649353] VFS: Disk quotas dquot_6.6.0 [ 0.650949] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.652507] *** VALIDATE ramfs *** [ 0.653285] *** VALIDATE hugetlbfs *** [ 0.654191] pnp: PnP ACPI init [ 0.655809] pnp: PnP ACPI: found 6 devices [ 0.669593] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.671679] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.672945] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.674263] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.675748] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.677147] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.679172] NET: Registered protocol family 2 [ 0.681398] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.685486] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.687727] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.691362] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.693384] TCP: Hash tables configured (established 65536 bind 65536) [ 0.695026] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.696813] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.698426] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700234] NET: Registered protocol family 1 [ 0.701886] RPC: Registered named UNIX socket transport module. [ 0.703247] RPC: Registered udp transport module. [ 0.704307] RPC: Registered tcp transport module. [ 0.705521] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.707186] NET: Registered protocol family 44 [ 0.708162] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.709492] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.710940] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.712514] PCI: CLS 0 bytes, default 64 [ 0.713691] Unpacking initramfs... [ 2.045886] debug: unmapping init [mem 0xffff987f7cc54000-0xffff987f7ffbffff] [ 2.048546] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.050618] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.053976] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.550837] Initialise system trusted keyrings [ 2.552784] Key type blacklist registered [ 2.554876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.562988] zbud: loaded [ 2.565280] *** VALIDATE nfs *** [ 2.566306] *** VALIDATE nfs4 *** [ 2.567503] pstore: using deflate compression [ 2.569516] Platform Keyring initialized [ 2.640522] NET: Registered protocol family 38 [ 2.641760] Key type asymmetric registered [ 2.642775] Asymmetric key parser 'x509' registered [ 2.644202] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.646179] io scheduler mq-deadline registered [ 2.647271] io scheduler kyber registered [ 2.648325] io scheduler bfq registered [ 2.649551] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.651854] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.654258] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.656799] ACPI: Power Button [PWRF] [ 2.662154] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.670227] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.680433] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.686235] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.698541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.726419] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.753937] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.759247] Non-volatile memory driver v1.3 [ 2.760868] Linux agpgart interface v0.103 [ 2.790483] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.793994] vda: detected capacity change from 0 to 68628480 [ 2.808718] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.812289] vdb: detected capacity change from 0 to 1073741824 [ 2.827060] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.829672] vdc: detected capacity change from 0 to 2621440000 [ 2.840915] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.843185] vdd: detected capacity change from 0 to 2621440000 [ 2.855719] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.859082] vde: detected capacity change from 0 to 4294967296 [ 2.872660] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.876287] vdf: detected capacity change from 0 to 4294967296 [ 2.880273] libphy: Fixed MDIO Bus: probed [ 2.883776] usbcore: registered new interface driver usbserial_generic [ 2.885495] usbserial: USB Serial support registered for generic [ 2.886984] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.889800] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.890968] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.892608] mousedev: PS/2 mouse device common for all mice [ 2.894667] rtc_cmos 00:05: RTC can wake from S4 [ 2.896512] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.897644] rtc_cmos 00:05: registered as rtc0 [ 2.900689] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.900947] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.902919] intel_pstate: CPU model not supported [ 2.907068] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.909566] hid: raw HID events driver (C) Jiri Kosina [ 2.911525] usbcore: registered new interface driver usbhid [ 2.913306] usbhid: USB HID core driver [ 2.914558] drop_monitor: Initializing network drop monitor service [ 2.916055] Initializing XFRM netlink socket [ 2.918222] NET: Registered protocol family 10 [ 2.920354] Segment Routing with IPv6 [ 2.921788] NET: Registered protocol family 17 [ 2.923894] mpls_gso: MPLS GSO support [ 2.928592] RAS: Correctable Errors collector initialized. [ 2.931030] AVX version of gcm_enc/dec engaged. [ 2.932862] AES CTR mode by8 optimization enabled [ 2.994511] sched_clock: Marking stable (2994488787, 0)->(3705602517, -711113730) [ 2.996447] registered taskstats version 1 [ 2.997962] Loading compiled-in X.509 certificates [ 2.999105] zswap: loaded using pool lzo/zbud [ 3.014041] Key type big_key registered [ 3.021795] Key type encrypted registered [ 3.022754] ima: No TPM chip found, activating TPM-bypass! [ 3.023806] ima: Allocated hash algorithm: sha1 [ 3.024774] ima: No architecture policies found [ 3.025741] evm: Initialising EVM extended attributes: [ 3.026800] evm: security.selinux [ 3.027599] evm: security.ima [ 3.028175] evm: security.capability [ 3.028848] evm: HMAC attrs: 0x1 [ 3.030285] rtc_cmos 00:05: setting system clock to 2026-01-02 20:06:20 UTC (1767384380) [ 3.034199] debug: unmapping init [mem 0xffffffffad003000-0xffffffffad1fffff] [ 3.035840] debug: unmapping init [mem 0xffffffffabd82000-0xffffffffac058fff] [ 3.042094] Write protecting the kernel read-only data: 28672k [ 3.044357] debug: unmapping init [mem 0xffffffffaa403000-0xffffffffaa5fffff] [ 3.046112] debug: unmapping init [mem 0xffffffffaad14000-0xffffffffaadfffff] [ 3.076886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.084407] systemd[1]: Detected virtualization kvm. [ 3.086099] systemd[1]: Detected architecture x86-64. [ 3.087638] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.108077] systemd[1]: No hostname configured. [ 3.109718] systemd[1]: Set hostname to . [ 3.111711] random: systemd: uninitialized urandom read (16 bytes read) [ 3.114022] systemd[1]: Initializing machine ID from random generator. [ 3.235226] random: systemd: uninitialized urandom read (16 bytes read) [ 3.238096] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.241351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.244497] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.248413] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.757728] device-mapper: uevent: version 1.0.3 [ 3.759081] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 3.956651] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.264315] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.318260] scsi host0: ata_piix [ 4.345180] scsi host1: ata_piix [ 4.346252] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.347847] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.938285] dracut-initqueue[579]: RTNETLINK answers: File exists [ 8.535091] random: crng init done [ 8.536214] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.035801] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.918693] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.115352] SELinux: Disabled at runtime. [ 10.171779] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.178490] systemd[1]: Detected virtualization kvm. [ 10.179843] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.615105] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.618140] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.621945] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.626179] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.630179] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.635093] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.638708] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 10.733963] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 11.047489] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.290710] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.331921] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.417166] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.423158] EDAC sbridge: Ver: 1.1.2 [ 12.686161] Key type dns_resolver registered [ 12.960829] NFS: Registering the id_resolver key type [ 12.962097] Key type id_resolver registered [ 12.963220] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-server login: [ 35.060505] libcfs: loading out-of-tree module taints kernel. [ 35.075390] Key type ._llcrypt registered [ 35.076336] Key type .llcrypt registered [ 35.113976] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 42.310532] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 42.979351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 42.986872] alg: No test for adler32 (adler32-zlib) [ 43.975989] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 44.265263] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 45.871194] Key type lgssc registered [ 46.509211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 52.416620] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 53.016556] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.029954] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 58.945807] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.526163] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.606029] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 70.792307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.811215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 70.817076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 71.895751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 71.905767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 71.940910] Lustre: lustre-MDT0000: new disk, initializing [ 71.962685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 71.968232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 73.236549] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 78.234329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.259189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 78.286312] Lustre: 6478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 78.297641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 78.299389] Lustre: Skipped 1 previous similar message [ 78.336239] Lustre: lustre-MDT0001: new disk, initializing [ 78.355135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.362206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 78.365373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.619167] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 81.843748] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 84.942166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.967179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 85.051090] Lustre: lustre-OST0000: new disk, initializing [ 85.051969] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 85.052772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 85.070667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 86.901739] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.099773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 90.105165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 90.117647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 92.471319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.507922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 92.558708] Lustre: lustre-OST0001: new disk, initializing [ 92.560795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 92.578163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.389359] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.770684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 97.773129] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 97.792366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 100.258480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 102.814441] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 109.627717] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing check_logdir /tmp/testlogs/ [ 111.017102] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing yml_node [ 112.346900] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 113.279778] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 114.053225] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 114.636486] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 2 15:08:10 EST 2026 [ 122.274225] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 122.737711] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 134.623490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 134.626148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 134.630594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.719917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 138.720077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 138.723902] Lustre: Skipped 2 previous similar messages [ 138.725901] Lustre: Skipped 2 previous similar messages [ 140.608529] LustreError: 12182:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 140.640617] Lustre: server umount lustre-MDT0000 complete [ 142.294656] LustreError: 7410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384519 with bad export cookie 7082819480974236059 [ 142.295831] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.298416] LustreError: 7410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 143.839732] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.839897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 143.842952] Lustre: Skipped 1 previous similar message [ 143.845691] Lustre: Skipped 1 previous similar message [ 147.871630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.873416] Lustre: Skipped 1 previous similar message [ 148.419979] LustreError: 12384:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 148.422088] LustreError: 12384:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 148.463149] Lustre: server umount lustre-MDT0001 complete [ 164.319148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 164.339087] LustreError: 12586:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 164.340767] LustreError: 12586:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 164.356197] Lustre: server umount lustre-OST0000 complete [ 171.689277] LustreError: 12788:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 171.691361] LustreError: 12788:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 171.727382] Lustre: server umount lustre-OST0001 complete [ 173.880576] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 177.282332] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 180.551701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.523572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.094560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.364410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.185285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.216763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.328108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.342153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.386700] Lustre: lustre-MDT0000: new disk, initializing [ 190.413162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.419717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.678764] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 196.163720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.184940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 196.219035] Lustre: 14966:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.353501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.688296] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 201.058739] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 202.515139] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 205.578177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.602603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.699049] Lustre: lustre-OST0000: new disk, initializing [ 205.700367] Lustre: Skipped 1 previous similar message [ 205.701894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.703443] Lustre: Skipped 2 previous similar messages [ 205.736386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.738045] Lustre: Skipped 1 previous similar message [ 207.551606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.554456] Lustre: Skipped 1 previous similar message [ 207.556614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.605619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.049431] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 211.789541] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 213.166762] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 213.233259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 214.570043] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 214.649840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 217.325036] LustreError: 17903:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 217.326926] LustreError: 17903:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 217.361336] Lustre: server umount lustre-OST0000 complete [ 222.175839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 222.177892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.181399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 229.347877] LustreError: 18104:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 229.350516] LustreError: 18104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 229.380303] Lustre: server umount lustre-MDT0000 complete [ 230.744690] LustreError: 14957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384608 with bad export cookie 7082819480974237578 [ 230.749580] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 230.750917] LustreError: 14957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 241.165783] Lustre: server umount lustre-MDT0001 complete [ 242.530064] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:10:18 (1767384618) [ 243.427272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 244.051366] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:10:19 (1767384619) [ 246.542329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.701125] LustreError: 18977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 246.707034] LustreError: 18977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 246.724083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.039837] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 251.006192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 252.420443] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 254.226730] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 255.433268] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 257.779357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 259.787653] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 261.610823] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 281.568194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 281.571054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.577444] Lustre: Skipped 1 previous similar message [ 281.580295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 281.582280] Lustre: Skipped 4 previous similar messages [ 286.129635] LustreError: 21242:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 286.131836] LustreError: 21242:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 286.163473] Lustre: server umount lustre-OST0000 complete [ 292.320161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 292.323413] LustreError: Skipped 1 previous similar message [ 292.325152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.329432] Lustre: Skipped 2 previous similar messages [ 293.461861] Lustre: server umount lustre-MDT0000 complete [ 294.643857] LustreError: 21448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384672 with bad export cookie 7082819480974238460 [ 294.650379] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.650511] LustreError: 21448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 294.782523] Lustre: server umount lustre-MDT0001 complete [ 298.410805] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 299.397162] Key type lgssc unregistered [ 299.534378] LNet: 22180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.538318] LNetError: 22180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.552335] LNet: Removed LNI 192.168.203.105@tcp [ 299.895094] Key type .llcrypt unregistered [ 299.896075] Key type ._llcrypt unregistered [ 309.172220] Key type ._llcrypt registered [ 309.173841] Key type .llcrypt registered [ 309.223375] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 309.606945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.622850] alg: No test for adler32 (adler32-zlib) [ 310.516557] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 310.624805] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 312.215165] Key type lgssc registered [ 312.610335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 315.860721] loop: module loaded [ 315.903790] loop0: detected capacity change from 0 to 204800000 [ 316.021653] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.027568] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.176755] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.206283] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.747187] loop0: detected capacity change from 0 to 204800000 [ 318.820333] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.826671] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.832965] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.882606] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 320.780509] loop0: detected capacity change from 0 to 204800000 [ 322.586811] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.604667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 322.608142] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.681402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 323.696275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 323.734811] Lustre: lustre-MDT0000: new disk, initializing [ 323.759294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.766299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 325.083968] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 328.599221] loop1: detected capacity change from 0 to 204800000 [ 330.483616] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 330.503480] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 330.604304] Lustre: lustre-OST07c6: new disk, initializing [ 330.606647] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 330.607139] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 330.608524] Lustre: Skipped 1 previous similar message [ 330.632484] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 332.525322] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 335.855276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 335.857926] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 335.868030] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 339.771115] Lustre: 24316:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 339.774195] Lustre: 24316:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 339.776118] Lustre: 24316:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 339.778554] Lustre: 24316:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 339.781239] Lustre: 24316:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 339.783208] Lustre: 24316:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 357.962524] Lustre: 24316:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 357.966243] Lustre: 24316:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 357.969062] Lustre: 24316:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 357.972149] Lustre: 24316:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 357.975333] Lustre: 24316:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 357.978380] Lustre: 24316:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 361.443293] Lustre: 24316:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 361.446036] Lustre: 24316:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 361.448983] Lustre: 24316:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 361.453113] Lustre: 24316:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 361.457088] Lustre: 24316:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 361.459389] Lustre: 24316:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 361.463682] Lustre: 24316:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 361.467925] Lustre: 24316:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 361.470604] Lustre: 24316:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 361.473032] Lustre: 24316:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 361.477261] Lustre: 24316:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 361.480038] Lustre: 24316:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 364.532057] Lustre: 24316:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 364.535171] Lustre: 24316:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 364.537821] Lustre: 24316:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 364.542017] Lustre: 24316:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 364.546374] Lustre: 24316:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 364.550823] Lustre: 24316:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 364.555290] Lustre: 24316:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 364.559960] Lustre: 24316:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 364.564255] Lustre: 24316:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 364.568475] Lustre: 24316:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 364.572302] Lustre: 24316:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 364.576609] Lustre: 24316:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 371.053468] Lustre: 24315:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 371.055955] Lustre: 24315:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 371.057960] Lustre: 24315:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 371.060595] Lustre: 24315:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 371.062796] Lustre: 24315:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 371.065708] Lustre: 24315:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 371.067967] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 371.070138] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 371.071986] Lustre: 24315:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 371.073960] Lustre: 24315:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 371.076072] Lustre: 24315:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 371.077867] Lustre: 24315:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 381.071554] Lustre: 24315:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 381.075225] Lustre: 24315:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 381.078118] Lustre: 24315:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 381.080761] Lustre: 24315:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 381.083689] Lustre: 24315:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 381.086211] Lustre: 24315:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 381.088813] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 381.091823] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 381.094425] Lustre: 24315:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 381.096801] Lustre: 24315:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 381.099213] Lustre: 24315:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 381.101361] Lustre: 24315:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 422.881634] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.881982] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 422.890630] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 428.001594] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 428.464168] LustreError: 26530:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 428.517463] Lustre: server umount lustre-OST07c6 complete [ 432.562178] LustreError: 27030:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 432.564102] LustreError: 27030:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 432.612604] Lustre: server umount lustre-MDT0000 complete [ 437.776436] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 438.797161] Key type lgssc unregistered [ 438.965320] LNet: 27864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.967553] LNetError: 27864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.976311] LNet: Removed LNI 192.168.203.105@tcp [ 439.302224] Key type .llcrypt unregistered [ 439.303299] Key type ._llcrypt unregistered [ 446.155830] Key type ._llcrypt registered [ 446.156868] Key type .llcrypt registered [ 446.188674] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 446.474408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 446.480637] alg: No test for adler32 (adler32-zlib) [ 447.323703] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 447.399033] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 448.975142] Key type lgssc registered [ 449.335624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 451.833805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 451.839430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.991511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.277196] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 457.076392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 457.199976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.485394] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 460.505862] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 461.751818] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 464.261828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 464.393788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.671300] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 468.655574] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.730934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 486.882518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 486.882748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.884549] LustreError: Skipped 1 previous similar message [ 486.890539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.905873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.907384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.910287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.646074] LustreError: 31560:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 489.679915] Lustre: server umount lustre-OST0000 complete [ 493.026868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.026972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.029226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.029234] Lustre: Skipped 1 previous similar message [ 493.030431] Lustre: Skipped 1 previous similar message [ 497.290937] LustreError: 31762:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 497.293037] LustreError: 31762:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 497.334857] Lustre: server umount lustre-MDT0000 complete [ 498.144458] LustreError: 29816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.149267] LustreError: 29816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 498.670086] LustreError: 29073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384876 with bad export cookie 4550469026024286402 [ 498.673065] LustreError: 29073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 498.673562] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.741631] LustreError: 31963:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 498.743377] LustreError: 31963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 498.796615] Lustre: server umount lustre-MDT0001 complete [ 502.769779] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 503.862424] Key type lgssc unregistered [ 504.010699] LNet: 32493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.013586] LNetError: 32493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.022594] LNet: Removed LNI 192.168.203.105@tcp [ 504.371296] Key type .llcrypt unregistered [ 504.372350] Key type ._llcrypt unregistered [ 511.875764] Key type ._llcrypt registered [ 511.876953] Key type .llcrypt registered [ 511.916380] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 512.281748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 512.287836] alg: No test for adler32 (adler32-zlib) [ 513.149217] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 513.237905] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 514.823261] Key type lgssc registered [ 515.279770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 517.990224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 517.997323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.167551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.472960] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 523.350281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 523.489551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 524.800528] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 526.606691] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 527.758947] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 530.174358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 530.303087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.211920] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 533.989986] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 543.664926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 548.836110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.836643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.838359] LustreError: Skipped 1 previous similar message [ 548.839072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.842239] Lustre: Skipped 1 previous similar message [ 548.846769] Lustre: Skipped 1 previous similar message [ 550.324252] LustreError: 35990:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 550.383248] Lustre: server umount lustre-OST0000 complete [ 553.794160] LustreError: 36192:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 553.797281] LustreError: 36192:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 553.832374] Lustre: server umount lustre-MDT0000 complete [ 553.953451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.954357] LustreError: 33719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.956685] Lustre: Skipped 1 previous similar message [ 553.962076] LustreError: 33719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 555.150557] LustreError: 34439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384932 with bad export cookie 13302618060702517402 [ 555.154617] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.156929] LustreError: 34439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 555.239693] LustreError: 36393:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 555.242349] LustreError: 36393:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 555.298385] Lustre: server umount lustre-MDT0001 complete [ 558.810065] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 559.788666] Key type lgssc unregistered [ 559.912382] LNet: 36923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.915302] LNetError: 36923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.926461] LNet: Removed LNI 192.168.203.105@tcp [ 560.218483] Key type .llcrypt unregistered [ 560.219753] Key type ._llcrypt unregistered [ 568.834763] Key type ._llcrypt registered [ 568.835828] Key type .llcrypt registered [ 568.885888] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 569.395475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 569.402723] alg: No test for adler32 (adler32-zlib) [ 570.264270] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 570.354858] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 571.935183] Key type lgssc registered [ 572.318769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 575.054339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 575.060517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.249092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.494037] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 580.337167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 580.486143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 581.714754] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 583.465838] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 584.606443] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 586.896363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 587.042255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.001553] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 590.852796] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 601.385523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 611.296185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 611.299548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.305710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.811517] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.813186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.814946] Lustre: Skipped 1 previous similar message [ 611.818876] Lustre: Skipped 2 previous similar messages [ 614.852648] LustreError: 40562:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 614.886173] Lustre: server umount lustre-MDT0000 complete [ 616.142504] LustreError: 38320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384993 with bad export cookie 13662971824217134962 [ 616.145519] LustreError: 38320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 616.145829] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.695038] LustreError: 39670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.701879] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [ 616.702188] LustreError: 39670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 616.928940] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.814379] LustreError: 39670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.814716] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [ 621.818696] LustreError: 39670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 621.820347] Lustre: Skipped 1 previous similar message [ 622.274658] LustreError: 40764:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 622.276721] LustreError: 40764:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 622.325862] Lustre: server umount lustre-MDT0001 complete [ 640.555423] LustreError: 40986:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 640.557423] LustreError: 40986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 640.592822] Lustre: server umount lustre-OST0000 complete [ 645.895478] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 646.920567] Key type lgssc unregistered [ 647.055591] LNet: 41823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.059428] LNetError: 41823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.072682] LNet: Removed LNI 192.168.203.105@tcp [ 647.573172] Key type .llcrypt unregistered [ 647.575818] Key type ._llcrypt unregistered [ 656.678259] Key type ._llcrypt registered [ 656.679339] Key type .llcrypt registered [ 656.715941] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 657.037580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.078112] alg: No test for adler32 (adler32-zlib) [ 657.938062] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 658.032128] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 659.615128] Key type lgssc registered [ 659.978964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 662.603547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.609615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.773428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.096338] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 667.825790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 667.956271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 669.293852] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 671.083563] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 672.286389] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 674.506678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 674.624494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.557225] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 678.352762] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 688.995615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 698.336562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.341819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.351196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.138663] Lustre: lustre-OST0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 699.362564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.364274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.325590] LustreError: 45466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 702.363596] Lustre: server umount lustre-OST0000 complete [ 715.325654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.326574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.328763] Lustre: Skipped 1 previous similar message [ 715.330883] Lustre: Skipped 2 previous similar messages [ 720.352114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.353764] Lustre: Skipped 1 previous similar message [ 721.617609] LustreError: 45843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 721.621942] LustreError: 45843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 721.657748] Lustre: server umount lustre-MDT0000 complete [ 723.278364] LustreError: 44816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385100 with bad export cookie 11091239207278605594 [ 723.285919] LustreError: 44816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 723.288766] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.412825] LustreError: 46044:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 729.416822] LustreError: 46044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 729.479911] Lustre: server umount lustre-MDT0001 complete [ 733.767631] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 734.696498] Key type lgssc unregistered [ 734.831529] LNet: 46576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.835785] LNetError: 46576:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.849536] LNet: Removed LNI 192.168.203.105@tcp [ 735.212743] Key type .llcrypt unregistered [ 735.214939] Key type ._llcrypt unregistered [ 745.276140] Key type ._llcrypt registered [ 745.277315] Key type .llcrypt registered [ 745.322175] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 745.777854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.806196] alg: No test for adler32 (adler32-zlib) [ 746.656194] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 746.735683] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 748.327124] Key type lgssc registered [ 748.724997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 751.515399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.527757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.725479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.978607] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 757.684900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 757.811682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 759.154780] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 761.330209] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 762.898301] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 765.788298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 765.965839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.443687] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 771.005299] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 779.312857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 784.074793] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 785.443395] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 796.296951] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-0215h19m31s [ 796.969072] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 798.688356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 798.690905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 798.694918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.715085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.719134] Lustre: Skipped 4 previous similar messages [ 802.055522] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 803.977543] LustreError: 50532:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 804.026985] Lustre: server umount lustre-MDT0000 complete [ 805.333666] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 805.857470] LustreError: 47948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.869120] LustreError: 47948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 808.118601] LustreError: 47944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.386700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.447574] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.582883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.626222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 809.642565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 810.236194] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 811.403259] Lustre: DEBUG MARKER: Wait for df (35911) ... [ 811.859953] Lustre: DEBUG MARKER: done [ 812.147803] LustreError: 47929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.5@tcp arrived at 1767385189 with bad export cookie 1522679861319643503 [ 812.152429] LustreError: 47929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 814.048730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.048904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.051255] LustreError: Skipped 1 previous similar message [ 814.055192] Lustre: Skipped 2 previous similar messages [ 814.055855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.059560] Lustre: Skipped 2 previous similar messages [ 815.094198] LustreError: 51583:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 815.098103] LustreError: 51583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 815.153730] Lustre: server umount lustre-OST0000 complete [ 820.195927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 820.197421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.198298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.198311] Lustre: Skipped 2 previous similar messages [ 820.215456] Lustre: Skipped 2 previous similar messages [ 822.852341] LustreError: 51790:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 822.854189] LustreError: 51790:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 822.891102] Lustre: server umount lustre-MDT0000 complete [ 824.401377] LustreError: 50265:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385201 with bad export cookie 1522679861319643944 [ 824.404764] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 824.405164] LustreError: 50265:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 824.525744] Lustre: server umount lustre-MDT0001 complete [ 828.563609] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 829.998772] Key type lgssc unregistered [ 830.162552] LNet: 52522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.167533] LNetError: 52522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.179590] LNet: Removed LNI 192.168.203.105@tcp [ 830.634455] Key type .llcrypt unregistered [ 830.636371] Key type ._llcrypt unregistered [ 845.502596] Key type ._llcrypt registered [ 845.503845] Key type .llcrypt registered [ 845.546651] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 852.803085] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 853.161368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 853.207403] alg: No test for adler32 (adler32-zlib) [ 854.060659] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 854.139792] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 855.719151] Key type lgssc registered [ 856.278738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.372676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 862.976899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 865.663866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 868.142570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 871.240052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 871.276715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 871.284426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.390208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 872.403460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 872.449484] Lustre: lustre-MDT0000: new disk, initializing [ 872.476453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.484753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 874.174910] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 879.785244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 879.806237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 879.842236] Lustre: 56444:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 879.866572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 879.868641] Lustre: Skipped 1 previous similar message [ 879.904021] Lustre: lustre-MDT0001: new disk, initializing [ 879.934284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 879.946332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 879.950111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 881.555722] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 885.108833] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 886.668575] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 889.772077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 889.808076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 889.965468] Lustre: lustre-OST0000: new disk, initializing [ 889.967570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 889.998444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.534382] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 893.061674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 893.067100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 893.119928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 896.449125] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 898.467832] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 898.565835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 900.249541] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 900.355780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 903.652248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 903.653731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.655593] LustreError: Skipped 1 previous similar message [ 903.656624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 903.661733] Lustre: Skipped 1 previous similar message [ 903.670273] Lustre: Skipped 1 previous similar message [ 907.254215] LustreError: 59387:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 907.319499] Lustre: server umount lustre-OST0000 complete [ 910.816964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 910.820383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.826453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.889172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.890569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.146279] LustreError: 59588:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 915.150468] LustreError: 59588:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 915.190492] Lustre: server umount lustre-MDT0000 complete [ 917.035756] LustreError: 57224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385294 with bad export cookie 5298376529114470432 [ 917.043129] LustreError: 57224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 917.044181] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.134490] LustreError: 59789:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 917.137117] LustreError: 59789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 917.208199] Lustre: server umount lustre-MDT0001 complete [ 920.665518] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 15:21:36 (1767385296) [ 923.450585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.710434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.598673] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 928.738418] LustreError: 60459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.749236] LustreError: 60459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 929.617444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 929.817906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 931.488348] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 934.051347] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 935.754706] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 939.118310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 939.310600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.040410] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 944.494833] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 952.646240] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-0215h22m08s [ 953.211597] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 956.213345] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 957.382480] Lustre: DEBUG MARKER: Creating a test file and stat it [ 957.403839] LustreError: 60454:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 977.673468] Lustre: lustre-MDT0000: Client 4f55d093-ee51-4ad2-933b-ee8d12502465 (at 192.168.203.5@tcp) reconnecting [ 997.447128] LustreError: 60454:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 998.511588] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 999.733587] Lustre: DEBUG MARKER: done [ 1001.952526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1001.956110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.961899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1001.966229] Lustre: Skipped 1 previous similar message [ 1006.051850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.054456] Lustre: Skipped 2 previous similar messages [ 1007.729722] LustreError: 63466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1007.736289] LustreError: 63466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1007.790942] Lustre: server umount lustre-OST0000 complete [ 1011.171427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.182153] Lustre: Skipped 2 previous similar messages [ 1015.625479] LustreError: 63667:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1015.627711] LustreError: 63667:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1015.665263] Lustre: server umount lustre-MDT0000 complete [ 1017.248527] LustreError: 60458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1017.254252] LustreError: 60458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1017.263481] LustreError: 62851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385394 with bad export cookie 5298376529114471314 [ 1017.269392] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.404764] Lustre: server umount lustre-MDT0001 complete [ 1021.384441] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1022.435454] Key type lgssc unregistered [ 1022.582395] LNet: 64400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.584760] LNetError: 64400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.591578] LNet: Removed LNI 192.168.203.105@tcp [ 1022.951226] Key type .llcrypt unregistered [ 1022.953314] Key type ._llcrypt unregistered [ 1036.371960] Key type ._llcrypt registered [ 1036.372899] Key type .llcrypt registered [ 1036.408205] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 1042.909596] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1043.449610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1043.456951] alg: No test for adler32 (adler32-zlib) [ 1044.312253] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1044.397925] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1045.975279] Key type lgssc registered [ 1046.472106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.359607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1052.002530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1054.521626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1057.024590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.627758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.667549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1059.677201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.774658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1060.786132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1060.823596] Lustre: lustre-MDT0000: new disk, initializing [ 1060.848875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.854838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1061.979063] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1066.178858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1066.209536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1066.251286] Lustre: 68323:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1066.290600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1066.294450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1066.297511] Lustre: Skipped 1 previous similar message [ 1066.337282] Lustre: lustre-MDT0001: new disk, initializing [ 1066.373282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1066.386882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1066.390359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1067.601541] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1070.666399] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.832588] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1074.220299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.245074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1074.344015] Lustre: lustre-OST0000: new disk, initializing [ 1074.345746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1074.372352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.579160] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1079.503495] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1080.754593] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1080.840571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1080.847481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1080.896100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1081.906331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1083.179614] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1083.241704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1085.922970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1085.926796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.929433] LustreError: Skipped 1 previous similar message [ 1085.939883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1085.946812] Lustre: Skipped 1 previous similar message [ 1089.907154] LustreError: 71268:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1089.970135] Lustre: server umount lustre-OST0000 complete [ 1092.064658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1092.069965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.077965] Lustre: Skipped 1 previous similar message [ 1092.081573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.161571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.162343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.798426] LustreError: 71470:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1097.800410] LustreError: 71470:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1097.826597] Lustre: server umount lustre-MDT0000 complete [ 1099.343176] LustreError: 68314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385476 with bad export cookie 2989141011176456660 [ 1099.349621] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.350397] LustreError: 68314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1099.453938] LustreError: 71673:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1099.457390] LustreError: 71673:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1099.525531] Lustre: server umount lustre-MDT0001 complete [ 1103.411589] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 15:24:39 (1767385479) [ 1104.106707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1104.895280] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 15:24:40 (1767385480) [ 1109.671286] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1111.342079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.563256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.048539] Lustre: Failing over lustre-MDT0000 [ 1112.051771] LustreError: 72747:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1112.058787] LustreError: 72747:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1112.064846] LustreError: 72747:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1112.169145] LustreError: 72798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1112.173098] LustreError: 72798:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1112.230279] Lustre: server umount lustre-MDT0000 complete [ 1114.293618] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:24:50 (1767385490) [ 1117.246369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.405190] LustreError: 73366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.409989] LustreError: 73366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1117.422789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.684567] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1122.472185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1124.122033] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1126.013858] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1127.304877] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1129.962192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1130.120222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.124106] Lustre: Skipped 1 previous similar message [ 1132.372168] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1134.079249] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1147.129625] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1152.996850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.999786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.006044] Lustre: Skipped 1 previous similar message [ 1153.013433] Lustre: Skipped 3 previous similar messages [ 1155.474690] LustreError: 75729:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1155.480060] LustreError: 75729:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1155.517139] Lustre: server umount lustre-MDT0000 complete [ 1157.323366] LustreError: 73347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385534 with bad export cookie 2989141011176457822 [ 1157.330579] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1157.330888] LustreError: 73347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1157.486949] Lustre: server umount lustre-MDT0001 complete [ 1159.101403] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1161.411829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1165.579373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.687147] LustreError: 76777:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1166.692656] LustreError: 76777:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000098844a1c x1853237935268992/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767385544 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1166.705793] LustreError: 76777:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1167.264146] LustreError: 66537:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005cf06217 x1853237935269632/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1167.449816] LustreError: 74974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.497807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.539836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1169.247829] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1172.643233] LustreError: 74974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.647174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.647494] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1172.669279] Lustre: Skipped 1 previous similar message [ 1173.346323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1174.495173] Lustre: 66539:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767385535/real 1767385535] req@00000000771f9bc9 x1853237935269248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767385551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1175.120563] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1177.528912] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1177.631107] Lustre: 66540:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767385539/real 1767385539] req@00000000695d18c5 x1853237935269376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767385555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1178.601310] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1179.212850] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.615261] Lustre: 66541:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767385540/real 1767385540] req@00000000f97f767b x1853237935269504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767385556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1182.623144] Lustre: 66538:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767385544/real 1767385544] req@000000009250377c x1853237935269760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767385560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1183.717116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1183.717726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.723202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.731457] Lustre: Skipped 1 previous similar message [ 1187.726092] LustreError: 78195:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1187.730024] LustreError: 78195:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1187.771968] Lustre: server umount lustre-MDT0000 complete [ 1188.835283] LustreError: 76790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.845185] LustreError: 76790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1189.562978] LustreError: 73347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385567 with bad export cookie 2989141011176459712 [ 1189.567922] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1189.568225] LustreError: 73347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1189.713553] Lustre: server umount lustre-MDT0001 complete [ 1192.443184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1196.269655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.096215] LustreError: 66537:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000081da45c x1853237935289600/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1200.261692] LustreError: 74973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.294050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.296606] Lustre: Skipped 1 previous similar message [ 1200.326939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1201.874990] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1205.730905] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.739595] Lustre: Skipped 1 previous similar message [ 1205.953884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1207.761632] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1209.823182] Lustre: 66541:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767385571/real 1767385571] req@0000000056f377e8 x1853237935289216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767385587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1210.119195] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1211.369557] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1211.603365] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1214.881854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.885835] Lustre: Skipped 2 previous similar messages [ 1219.984297] LustreError: 80653:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1219.988486] LustreError: 80653:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1220.024771] Lustre: server umount lustre-MDT0000 complete [ 1220.064927] LustreError: 79251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.076559] LustreError: 79251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1221.771967] LustreError: 74085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385599 with bad export cookie 2989141011176461091 [ 1221.778406] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.779330] LustreError: 74085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1221.903410] Lustre: server umount lustre-MDT0001 complete [ 1224.682259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1226.307377] Lustre: DEBUG MARKER: files compared the same [ 1237.566136] Lustre: server umount lustre-OST0000 complete [ 1244.870396] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1246.024681] Key type lgssc unregistered [ 1246.141532] LNet: 82284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.146068] LNetError: 82284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.159502] LNet: Removed LNI 192.168.203.105@tcp [ 1246.574981] Key type .llcrypt unregistered [ 1246.577328] Key type ._llcrypt unregistered [ 1256.762396] Key type ._llcrypt registered [ 1256.763398] Key type .llcrypt registered [ 1256.798720] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1257.214683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.220160] alg: No test for adler32 (adler32-zlib) [ 1258.076055] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1258.174733] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1259.767190] Key type lgssc registered [ 1260.316138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1263.591388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1263.596733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.770759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.510116] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug +malloc all [ 1270.417350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1270.562349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1272.209343] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug +malloc all [ 1274.790494] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1276.248775] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1279.463739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1279.636396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.341772] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug +malloc all [ 1284.914547] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1289.894543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1295.330189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.330931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.336758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.341581] Lustre: Skipped 1 previous similar message [ 1299.376179] LustreError: 85906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1299.435647] Lustre: server umount lustre-OST0000 complete [ 1301.471858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.476674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.484017] Lustre: Skipped 1 previous similar message [ 1301.487175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.568393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.568734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.329496] LustreError: 86110:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1307.331432] LustreError: 86110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1307.363645] Lustre: server umount lustre-MDT0000 complete [ 1308.933204] LustreError: 85277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385686 with bad export cookie 18383465085137300584 [ 1308.935477] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.936133] LustreError: 85277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1309.018390] LustreError: 86313:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1309.022511] LustreError: 86313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1309.082746] Lustre: server umount lustre-MDT0001 complete [ 1313.920081] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1315.566489] Key type lgssc unregistered [ 1315.754747] LNet: 86845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.760211] LNetError: 86845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.772738] LNet: Removed LNI 192.168.203.105@tcp [ 1316.271445] Key type .llcrypt unregistered [ 1316.273968] Key type ._llcrypt unregistered [ 1327.530060] Key type ._llcrypt registered [ 1327.531481] Key type .llcrypt registered [ 1327.583698] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1327.978977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.988747] alg: No test for adler32 (adler32-zlib) [ 1328.850597] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1328.947748] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1330.535137] Key type lgssc registered [ 1331.150959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1334.262895] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1334.274604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1349.727420] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1354.847226] LustreError: 88192:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 1381.471240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.928954] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1386.219363] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1389.667612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.718885] Lustre: *** cfs_fail_loc=706, val=0*** [ 1389.902561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1389.940329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1391.243936] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1394.889921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1395.021511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1396.175752] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1398.410250] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1399.522521] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1400.288459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1400.288801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.290947] LustreError: Skipped 1 previous similar message [ 1400.294876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.297743] Lustre: Skipped 1 previous similar message [ 1401.313947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1401.316262] Lustre: Skipped 1 previous similar message [ 1406.258143] LustreError: 90523:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1406.316802] Lustre: server umount lustre-OST0000 complete [ 1410.528062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1410.532343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.538481] Lustre: Skipped 1 previous similar message [ 1410.541548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.554489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.223849] LustreError: 90726:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1414.228953] LustreError: 90726:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1414.260641] Lustre: server umount lustre-MDT0000 complete [ 1415.719410] LustreError: 90728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385793 with bad export cookie 7017190791447586702 [ 1415.722521] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.724134] LustreError: 90728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1415.793665] LustreError: 90930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1415.795778] LustreError: 90930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1415.848519] Lustre: server umount lustre-MDT0001 complete [ 1419.962366] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1420.890334] Key type lgssc unregistered [ 1421.028491] LNet: 91460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.032471] LNetError: 91460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.042423] LNet: Removed LNI 192.168.203.105@tcp [ 1421.384919] Key type .llcrypt unregistered [ 1421.386853] Key type ._llcrypt unregistered [ 1431.564627] Key type ._llcrypt registered [ 1431.565717] Key type .llcrypt registered [ 1431.599227] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1431.971786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.977718] alg: No test for adler32 (adler32-zlib) [ 1432.818508] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1432.894045] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1434.495155] Key type lgssc registered [ 1435.044990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1438.496448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1438.507791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.169251] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1444.234443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1444.538199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.074985] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1449.719840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1449.908442] LustreError: 93505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.923474] LustreError: 93505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1449.923538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.187337] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1453.987817] LustreError: 93509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.993390] LustreError: 93509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1455.617882] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.626911] Lustre: MGS: Client 968d5632-2bf7-4f72-b25e-fb8dccb76b36 (at 0@lo) reconnecting [ 1455.644894] LustreError: 93519:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1455.779544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.825862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1457.237408] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1466.340853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.343419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.347927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.350227] Lustre: Skipped 1 previous similar message [ 1471.458080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1471.459759] Lustre: Skipped 1 previous similar message [ 1471.725233] LustreError: 95474:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1471.760888] Lustre: server umount lustre-OST0000 complete [ 1476.577917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.578706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.587408] Lustre: Skipped 1 previous similar message [ 1476.590747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.593930] Lustre: Skipped 1 previous similar message [ 1479.569754] LustreError: 95677:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1479.574112] LustreError: 95677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1479.589679] Lustre: server umount lustre-MDT0000 complete [ 1481.541504] LustreError: 95879:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1481.545128] LustreError: 95879:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1481.594985] Lustre: server umount lustre-MDT0001 complete [ 1483.102684] Lustre: server umount MGS complete [ 1488.081824] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:31:03 (1767385863) [ 1500.234365] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 1504.688143] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1508.120546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.738468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1513.332903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1516.298289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1518.925246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1518.950962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.290319] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1525.408319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1525.447952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1525.573400] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1525.600413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1525.658840] Lustre: lustre-MDT0001: new disk, initializing [ 1525.701065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1525.717677] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1525.722557] Lustre: 100587:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1525.728326] LustreError: 100587:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1527.534699] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1531.973745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.010769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1532.160822] Lustre: lustre-OST0000: new disk, initializing [ 1532.166493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1532.204525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.920729] Lustre: 101428:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000003fd63575 x1853238343140224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767385927 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1534.150946] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1537.976650] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1540.449567] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1540.453401] Lustre: MGS: Client 94ce4a96-32e6-48f8-a935-67d94404f0b2 (at 0@lo) reconnecting [ 1540.468329] Lustre: 99871:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1540.477535] LustreError: 100573:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1540.479979] LustreError: 100573:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1540.485214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1540.519285] Lustre: lustre-MDT0000: new disk, initializing [ 1540.567358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.575237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1541.599617] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1541.602098] Lustre: 102299:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a6a27ffe x1853238343146240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767385935 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1541.724590] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1548.258176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1548.260689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1550.689199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1550.691862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1550.754918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1571.297285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.304068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.314628] Lustre: Skipped 1 previous similar message [ 1571.319597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1577.011518] LustreError: 102825:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1577.015886] LustreError: 102825:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1577.070202] Lustre: server umount lustre-OST0000 complete [ 1583.074831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.075102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1583.076251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1583.076260] Lustre: Skipped 2 previous similar messages [ 1583.082824] Lustre: Skipped 2 previous similar messages [ 1593.311192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1593.313754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.322454] Lustre: Skipped 4 previous similar messages [ 1593.379348] LustreError: 103027:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1593.383608] LustreError: 103027:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1593.396143] Lustre: server umount lustre-MDT0000 complete [ 1601.369950] Lustre: server umount lustre-MDT0001 complete [ 1603.168006] LustreError: 103433:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1603.173090] LustreError: 103433:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1603.210542] Lustre: server umount MGS complete [ 1607.882722] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:33:03 (1767385983) [ 1615.153400] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1616.599992] Key type lgssc unregistered [ 1616.767547] LNet: 104769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.772703] LNetError: 104769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.781638] LNet: Removed LNI 192.168.203.105@tcp [ 1617.238301] Key type .llcrypt unregistered [ 1617.240590] Key type ._llcrypt unregistered [ 1625.294790] Key type ._llcrypt registered [ 1625.295778] Key type .llcrypt registered [ 1625.334007] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1625.673078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.700054] alg: No test for adler32 (adler32-zlib) [ 1626.553611] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1626.631100] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1628.207175] Key type lgssc registered [ 1628.685346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.387586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.399391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.596173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.149474] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1639.005289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1639.181094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1640.811804] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1642.965723] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1644.653824] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1647.865964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1648.051751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1650.457358] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1653.100359] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1659.307755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1669.605725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1669.605799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.608062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.622608] Lustre: Skipped 1 previous similar message [ 1672.882186] LustreError: 108225:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1672.938665] Lustre: server umount lustre-OST0000 complete [ 1674.722189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.723207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.728302] Lustre: Skipped 1 previous similar message [ 1674.732159] Lustre: Skipped 2 previous similar messages [ 1679.841399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.846267] Lustre: Skipped 1 previous similar message [ 1680.841386] LustreError: 108427:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1680.844395] LustreError: 108427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1680.881470] Lustre: server umount lustre-MDT0000 complete [ 1682.492551] LustreError: 105935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386059 with bad export cookie 15466466803443255394 [ 1682.500111] LustreError: 105935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1682.501349] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1682.592794] LustreError: 108629:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1682.596824] LustreError: 108629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1682.666418] Lustre: server umount lustre-MDT0001 complete [ 1687.510733] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1689.069614] Key type lgssc unregistered [ 1689.258810] LNet: 109160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.263558] LNetError: 109160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.274784] LNet: Removed LNI 192.168.203.105@tcp [ 1689.717176] Key type .llcrypt unregistered [ 1689.718975] Key type ._llcrypt unregistered [ 1697.918578] Key type ._llcrypt registered [ 1697.920112] Key type .llcrypt registered [ 1697.970084] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1698.347711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.363103] alg: No test for adler32 (adler32-zlib) [ 1699.268427] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1699.407887] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1701.023189] Key type lgssc registered [ 1701.647595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.171243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.171491] LustreError: 110200:0:(tgt_mount.c:2528:server_fill_super()) cfs_race id 716 sleeping [ 1704.176391] Lustre: Skipped 1 previous similar message [ 1704.176408] LustreError: 110201:0:(tgt_mount.c:2528:server_fill_super()) cfs_fail_race id 716 waking [ 1704.188149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.191793] LustreError: 110200:0:(tgt_mount.c:2528:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1704.204882] LustreError: 110200:0:(tgt_mount.c:2540:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1704.213355] LustreError: 110200:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1705.418572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.192831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1709.360954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1710.888432] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1713.048714] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing lsmod [ 1714.446210] LustreError: 111376:0:(tgt_mount.c:2528:server_fill_super()) cfs_race id 716 sleeping [ 1714.448154] LustreError: 111377:0:(tgt_mount.c:2528:server_fill_super()) cfs_fail_race id 716 waking [ 1714.456205] LustreError: 111376:0:(tgt_mount.c:2528:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1714.461070] LustreError: 111376:0:(tgt_mount.c:2540:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1714.466357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1714.468338] LustreError: 111376:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1714.637474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.687497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1719.015156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.019544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1719.024117] Lustre: Skipped 2 previous similar messages [ 1719.029075] Lustre: Skipped 1 previous similar message [ 1722.129183] LustreError: 111569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1722.172359] Lustre: server umount lustre-MDT0000 complete [ 1723.948518] LustreError: 110216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386101 with bad export cookie 3659153589080313626 [ 1723.954358] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1723.955520] LustreError: 110216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1724.384849] LustreError: 110675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1724.385189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.394488] LustreError: 110675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1724.404724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1725.151209] LustreError: 110674:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000933391d1 x1853238622044800/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1725.172228] LustreError: 111774:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1725.176482] LustreError: 111774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1725.245848] Lustre: server umount lustre-MDT0001 complete [ 1726.839901] LustreError: 111974:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1726.844640] LustreError: 111974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1726.908472] Lustre: server umount lustre-OST0000 complete [ 1730.812562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.026743] LustreError: 112450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.065872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.736675] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1736.160559] LustreError: 112450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.550701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.715729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.304891] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1740.931471] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1742.484546] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1745.446605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1745.608698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.936444] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1750.510322] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1755.881118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 1761.253757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1761.255260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.259927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.267588] Lustre: Skipped 1 previous similar message [ 1761.275758] Lustre: Skipped 1 previous similar message [ 1763.249157] LustreError: 114696:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1763.252994] LustreError: 114696:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1763.306771] Lustre: server umount lustre-OST0000 complete [ 1766.370752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.371796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.380047] Lustre: Skipped 1 previous similar message [ 1770.769093] LustreError: 114897:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1770.773271] LustreError: 114897:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1770.806087] Lustre: server umount lustre-MDT0000 complete [ 1771.488782] LustreError: 112450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.499338] LustreError: 112450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1772.548794] LustreError: 112431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386150 with bad export cookie 3659153589080314536 [ 1772.554840] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.555361] LustreError: 112431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1772.702461] Lustre: server umount lustre-MDT0001 complete [ 1777.428114] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1778.671579] Key type lgssc unregistered [ 1778.805563] LNet: 115629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.811840] LNetError: 115629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.821633] LNet: Removed LNI 192.168.203.105@tcp [ 1779.284589] Key type .llcrypt unregistered [ 1779.286934] Key type ._llcrypt unregistered [ 1790.308691] Key type ._llcrypt registered [ 1790.310137] Key type .llcrypt registered [ 1790.347088] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1790.669098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.698991] alg: No test for adler32 (adler32-zlib) [ 1791.560564] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1791.659826] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1793.247192] Key type lgssc registered [ 1793.792138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.206090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.217698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.444972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.967708] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1803.825965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1803.951362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1805.459774] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1807.902332] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1809.365310] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1812.664103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1812.845684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.940674] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1817.078636] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1823.080031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1829.465029] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1830.205206] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1833.440714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.442162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.443773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.447452] Lustre: Skipped 1 previous similar message [ 1833.450747] Lustre: Skipped 1 previous similar message [ 1837.556528] LustreError: 119372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1837.589836] Lustre: server umount lustre-OST0000 complete [ 1840.095882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1840.099529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.105394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.680184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.681618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.990646] LustreError: 119577:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1844.993067] LustreError: 119577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1845.022201] Lustre: server umount lustre-MDT0000 complete [ 1846.242096] LustreError: 119225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386223 with bad export cookie 2487588014632570691 [ 1846.245044] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.246699] LustreError: 119225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1846.311758] LustreError: 119779:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1846.314266] LustreError: 119779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1846.361111] Lustre: server umount lustre-MDT0001 complete [ 1850.167813] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1851.101921] Key type lgssc unregistered [ 1851.236497] LNet: 120309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.240189] LNetError: 120309:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.247509] LNet: Removed LNI 192.168.203.105@tcp [ 1851.561041] Key type .llcrypt unregistered [ 1851.562259] Key type ._llcrypt unregistered [ 1859.868192] Key type ._llcrypt registered [ 1859.869623] Key type .llcrypt registered [ 1859.925542] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1860.302156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.340090] alg: No test for adler32 (adler32-zlib) [ 1861.184943] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1861.272508] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1862.871140] Key type lgssc registered [ 1863.272356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1866.363657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1866.369802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.593720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.034983] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1872.513913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1872.672607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.219535] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1876.426241] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1877.985538] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1880.852659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1881.016886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.323748] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1885.382494] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1893.294672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1903.587253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.592133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.595856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.599566] Lustre: Skipped 1 previous similar message [ 1905.459203] LustreError: 123755:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1905.521480] Lustre: server umount lustre-OST0000 complete [ 1908.707632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.707780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1908.709184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.709190] Lustre: Skipped 1 previous similar message [ 1908.712411] Lustre: Skipped 1 previous similar message [ 1908.715784] LustreError: Skipped 1 previous similar message [ 1913.418607] LustreError: 123959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1913.424683] LustreError: 123959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1913.471346] Lustre: server umount lustre-MDT0000 complete [ 1913.824875] LustreError: 122703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.832928] LustreError: 122703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1915.274336] LustreError: 121472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386292 with bad export cookie 17337280381877475244 [ 1915.278192] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.282145] LustreError: 121472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1915.391629] LustreError: 124162:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1915.395583] LustreError: 124162:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1915.463607] Lustre: server umount lustre-MDT0001 complete [ 1919.532301] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1920.642320] Key type lgssc unregistered [ 1920.764575] LNet: 124693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.766793] LNetError: 124693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.777599] LNet: Removed LNI 192.168.203.105@tcp [ 1921.255237] Key type .llcrypt unregistered [ 1921.257634] Key type ._llcrypt unregistered [ 1931.635392] Key type ._llcrypt registered [ 1931.638015] Key type .llcrypt registered [ 1931.690601] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1932.213605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.244420] alg: No test for adler32 (adler32-zlib) [ 1933.151103] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1933.275616] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1934.887198] Key type lgssc registered [ 1935.529632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.730971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1938.738270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.920397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.726375] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1945.941383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1946.093076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1947.898262] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1949.899869] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.171542] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1954.473359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1954.684423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.366208] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1959.571722] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1964.917297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 1973.017767] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1974.807982] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1974.813231] Lustre: Skipped 2 previous similar messages [ 1976.887567] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1976.892550] Lustre: Skipped 2 previous similar messages [ 1979.403640] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1979.420779] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1985.840536] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1985.847230] Lustre: Skipped 2 previous similar messages [ 1993.737217] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.5@tcp 192.168.0.[2,10]@tcp [ 1996.403144] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.5/24@tcp in log lustre-MDT0000 [ 1996.409152] Lustre: Skipped 2 previous similar messages [ 2003.986416] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.5/24@tcp [ 2003.990444] Lustre: Skipped 1 previous similar message [ 2005.989862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.990872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.999077] Lustre: Skipped 1 previous similar message [ 2006.006719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.107593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.111624] Lustre: Skipped 2 previous similar messages [ 2011.826192] LustreError: 129983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2011.883820] Lustre: server umount lustre-OST0000 complete [ 2016.226527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.227629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.234423] Lustre: Skipped 1 previous similar message [ 2019.856890] LustreError: 130184:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2019.860862] LustreError: 130184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2019.895048] Lustre: server umount lustre-MDT0000 complete [ 2021.345746] LustreError: 127237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.355924] LustreError: 127237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2021.664136] LustreError: 128390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386399 with bad export cookie 6654419107485379373 [ 2021.669578] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.671582] LustreError: 128390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2021.766566] LustreError: 130386:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2021.769475] LustreError: 130386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2021.825442] Lustre: server umount lustre-MDT0001 complete [ 2026.597814] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2028.114325] Key type lgssc unregistered [ 2028.290701] LNet: 130917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.295900] LNetError: 130917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.311536] LNet: Removed LNI 192.168.203.105@tcp [ 2028.765847] Key type .llcrypt unregistered [ 2028.768254] Key type ._llcrypt unregistered [ 2042.418553] Key type ._llcrypt registered [ 2042.420524] Key type .llcrypt registered [ 2042.479889] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2042.932898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.940588] alg: No test for adler32 (adler32-zlib) [ 2043.806961] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2043.906064] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2045.511173] Key type lgssc registered [ 2046.121848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2049.717395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2049.725948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.974715] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2050.979797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.987242] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.5/24@tcp [ 2052.864704] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2057.051296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2057.225371] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2057.241428] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.5/24@tcp [ 2057.275241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.213876] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2061.805425] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2063.499457] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2066.815179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2067.010356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.726449] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2072.314107] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2076.279687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2081.763351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2081.767124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.778674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.782646] Lustre: Skipped 1 previous similar message [ 2086.888290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.892312] Lustre: Skipped 1 previous similar message [ 2087.538054] LustreError: 134738:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2087.593717] Lustre: server umount lustre-OST0000 complete [ 2092.004457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.005523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.009755] Lustre: Skipped 2 previous similar messages [ 2092.016321] Lustre: Skipped 1 previous similar message [ 2095.303534] LustreError: 134939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2095.308123] LustreError: 134939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2095.350379] Lustre: server umount lustre-MDT0000 complete [ 2096.903305] LustreError: 132417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386474 with bad export cookie 11003197826635661921 [ 2096.910270] LustreError: 132417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2096.910439] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.008277] LustreError: 135140:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2097.010397] LustreError: 135140:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2097.097853] Lustre: server umount lustre-MDT0001 complete [ 2101.970933] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2103.555085] Key type lgssc unregistered [ 2103.750746] LNet: 135672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.755776] LNetError: 135672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.766832] LNet: Removed LNI 192.168.203.105@tcp [ 2104.261618] Key type .llcrypt unregistered [ 2104.263657] Key type ._llcrypt unregistered [ 2115.539677] Key type ._llcrypt registered [ 2115.541412] Key type .llcrypt registered [ 2115.613905] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2123.266828] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2123.646482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.681075] alg: No test for adler32 (adler32-zlib) [ 2124.597568] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2124.732254] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2126.343201] Key type lgssc registered [ 2126.995400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2131.395379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2134.325619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2137.049341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2139.898677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2143.211796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2143.256175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2143.268119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.392365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2144.411374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2144.469247] Lustre: lustre-MDT0000: new disk, initializing [ 2144.496996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.507775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2146.293717] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2151.664685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2151.709527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2151.769704] Lustre: 139018:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2151.823952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2151.829195] Lustre: Skipped 1 previous similar message [ 2151.885194] Lustre: lustre-MDT0001: new disk, initializing [ 2151.966338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.994278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2152.001324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2153.772628] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2157.496326] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2159.185901] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2162.224144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2162.263919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2162.430568] Lustre: lustre-OST0000: new disk, initializing [ 2162.433190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2162.494617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.568169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2162.570910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2162.581381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2164.999577] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2168.952120] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2170.798861] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2170.928941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2172.518233] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2172.595744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2178.020469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.021539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.031680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.039147] Lustre: Skipped 1 previous similar message [ 2179.629116] LustreError: 141962:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2179.674164] Lustre: server umount lustre-OST0000 complete [ 2182.624728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.630671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.639819] Lustre: Skipped 1 previous similar message [ 2182.643790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.538025] LustreError: 142164:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2187.541792] LustreError: 142164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2187.579080] Lustre: server umount lustre-MDT0000 complete [ 2189.319081] LustreError: 139011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386566 with bad export cookie 12585281612758786755 [ 2189.325328] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.329029] LustreError: 139011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2189.435914] LustreError: 142367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2189.440266] LustreError: 142367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2189.504388] Lustre: server umount lustre-MDT0001 complete [ 2193.263444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.485269] LustreError: 142842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.495039] LustreError: 142842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2193.518328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2195.197484] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2199.008911] LustreError: 142842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.042325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2199.227805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.902964] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2203.397680] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2205.145686] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2208.418821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2208.574038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.991689] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2213.574568] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2215.405479] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2218.636304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2220.618866] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2220.752263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2225.028232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2225.073655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2225.177537] Lustre: lustre-OST0001: new disk, initializing [ 2225.181918] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2225.237066] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.711494] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2231.638484] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2232.356156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2232.365468] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2232.410711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2232.775595] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2234.369742] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2234.479762] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2235.981077] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f5284df1800.ost_server_uuid 50 [ 2236.733784] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f5284df1800.ost_server_uuid in FULL state after 0 sec [ 2242.529779] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.532230] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.533661] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2242.536390] Lustre: Skipped 2 previous similar messages [ 2242.537901] Lustre: Skipped 2 previous similar messages [ 2245.237695] LustreError: 147119:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2245.242104] LustreError: 147119:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2245.271934] Lustre: server umount lustre-OST0001 complete [ 2247.652096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.654578] LustreError: 144439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.655796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2247.655810] Lustre: Skipped 1 previous similar message [ 2247.660281] Lustre: Skipped 1 previous similar message [ 2247.670896] LustreError: 144439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2252.770967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.772366] LustreError: 144438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.773625] Lustre: Skipped 2 previous similar messages [ 2253.106202] LustreError: 147321:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2253.110830] LustreError: 147321:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2253.169507] Lustre: server umount lustre-OST0000 complete [ 2255.840454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2255.845990] LustreError: Skipped 1 previous similar message [ 2255.849508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.101768] Lustre: server umount lustre-MDT0000 complete [ 2262.881953] LustreError: 144461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386640 with bad export cookie 12585281612758787623 [ 2262.887751] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.889170] LustreError: 144461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2262.992901] LustreError: 147726:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2262.995549] LustreError: 147726:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2263.048644] Lustre: server umount lustre-MDT0001 complete [ 2270.579232] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2271.730407] Key type lgssc unregistered [ 2271.849378] LNet: 148713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.851547] LNetError: 148713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.862257] LNet: Removed LNI 192.168.203.105@tcp [ 2272.169808] Key type .llcrypt unregistered [ 2272.171949] Key type ._llcrypt unregistered [ 2283.885436] Key type ._llcrypt registered [ 2283.886482] Key type .llcrypt registered [ 2283.919878] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2290.658663] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2291.032878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.041450] alg: No test for adler32 (adler32-zlib) [ 2291.881871] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2291.959805] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2293.535176] Key type lgssc registered [ 2294.055695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.517700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2300.393893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2302.611303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2305.753407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2308.856377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2308.892394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.901400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.020863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2310.037580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2310.089098] Lustre: lustre-MDT0000: new disk, initializing [ 2310.124801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.137188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2311.576682] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2316.404423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2316.437516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2316.479248] Lustre: 152637:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2316.507754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2316.510106] Lustre: Skipped 1 previous similar message [ 2316.548061] Lustre: lustre-MDT0001: new disk, initializing [ 2316.581312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2316.596692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2316.600712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2318.151789] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2321.656816] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2322.733248] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2325.275368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2325.301094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2325.408691] Lustre: lustre-OST0000: new disk, initializing [ 2325.410501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2325.439712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.197915] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2327.821152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2327.828118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2327.882742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2330.508026] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2331.999359] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2332.060413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2333.214288] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2333.291273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2336.178202] LustreError: 155577:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2336.239918] Lustre: server umount lustre-OST0000 complete [ 2338.273416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.274167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.279385] Lustre: Skipped 1 previous similar message [ 2343.392893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.396440] Lustre: Skipped 2 previous similar messages [ 2348.513332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.517574] Lustre: Skipped 1 previous similar message [ 2352.095050] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2352.131452] LustreError: 155777:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2352.133578] LustreError: 155777:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2352.162058] Lustre: server umount lustre-MDT0000 complete [ 2353.302237] LustreError: 152627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386730 with bad export cookie 7048257122106470613 [ 2353.305449] LustreError: 152627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2353.306254] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.853504] LustreError: 155979:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2362.855326] LustreError: 155979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2362.896494] Lustre: server umount lustre-MDT0001 complete [ 2365.823529] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 15:45:41 (1767386741) [ 2371.763513] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2374.539503] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2378.611820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2381.791659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.447567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2387.052882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2390.391579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2390.413567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.502510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2390.512432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2390.552247] Lustre: lustre-MDT0000: new disk, initializing [ 2390.580351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.586870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2392.062841] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2396.315209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2396.336199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2396.364833] Lustre: 158966:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2396.384257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2396.385815] Lustre: Skipped 1 previous similar message [ 2396.419895] Lustre: lustre-MDT0001: new disk, initializing [ 2396.447705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.457332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2396.460074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2397.522803] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2401.004522] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2402.760816] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2404.955070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.979075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2405.081957] Lustre: lustre-OST0000: new disk, initializing [ 2405.083621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2405.111855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.755179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2405.757990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2405.794786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2406.805401] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2409.673540] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2411.510111] Lustre: Failing over lustre-OST0000 [ 2411.542135] LustreError: 161405:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2411.543862] LustreError: 161405:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2411.576890] Lustre: server umount lustre-OST0000 complete [ 2424.781233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2424.932533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.937076] Lustre: Skipped 1 previous similar message [ 2424.942244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.949294] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2426.120685] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2427.231721] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2428.810428] Lustre: Failing over lustre-MDT0000 [ 2428.887988] LustreError: 162378:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2428.891163] LustreError: 162378:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2428.926229] Lustre: server umount lustre-MDT0000 complete [ 2430.436242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.444211] Lustre: Skipped 2 previous similar messages [ 2432.479288] Lustre: 150851:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767386793/real 1767386793] req@000000001477b82d x1853239243890304/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767386809 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2432.495837] Lustre: 150851:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2435.552953] LustreError: 159751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.562742] LustreError: 159751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2436.343642] LustreError: 158972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.575245] Lustre: 150852:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767386798/real 1767386798] req@00000000af44a69d x1853239243892096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767386814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2440.673146] LustreError: 159602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.678154] LustreError: 159602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2442.821114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.859865] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2442.864836] Lustre: 158969:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.203.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2442.945350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2442.969491] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2442.969498] Lustre: lustre-OST0000: Recovery over after 0:16, of 3 clients 3 recovered and 0 were evicted. [ 2443.005396] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2444.510080] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2446.584299] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2448.368505] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2448.371790] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2448.373950] Lustre: Skipped 1 previous similar message [ 2448.417950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2453.478499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.478979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2453.487654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.491175] LustreError: Skipped 1 previous similar message [ 2455.663661] LustreError: 163264:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2455.668251] LustreError: 163264:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2455.724406] Lustre: server umount lustre-OST0000 complete [ 2458.594093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2458.594722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.609103] Lustre: Skipped 1 previous similar message [ 2458.613414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.617491] Lustre: Skipped 2 previous similar messages [ 2468.833001] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.837091] Lustre: Skipped 3 previous similar messages [ 2471.391071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2471.444932] Lustre: server umount lustre-MDT0000 complete [ 2473.077803] LustreError: 158957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386850 with bad export cookie 7048257122106472678 [ 2473.082459] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.085359] LustreError: 158957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2473.197069] LustreError: 163671:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2473.201028] LustreError: 163671:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2473.260234] Lustre: server umount lustre-MDT0001 complete [ 2477.760226] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2479.279086] Key type lgssc unregistered [ 2479.451608] LNet: 164202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2479.456638] LNetError: 164202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.467573] LNet: Removed LNI 192.168.203.105@tcp [ 2479.833492] Key type .llcrypt unregistered [ 2479.834753] Key type ._llcrypt unregistered [ 2501.823648] Key type ._llcrypt registered [ 2501.824673] Key type .llcrypt registered [ 2501.855234] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2502.148987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.186219] alg: No test for adler32 (adler32-zlib) [ 2503.041078] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2503.130810] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2504.703206] Key type lgssc registered [ 2505.242920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2508.613778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2508.623969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.810681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2511.275371] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2515.138465] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2517.498325] LustreError: 165729:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2517.504588] LustreError: 165729:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2517.510601] LustreError: 165729:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2517.602705] LustreError: 166251:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2517.653246] Lustre: server umount lustre-MDT0000 complete [ 2523.740589] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2527.791143] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2532.111572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2534.845258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2537.521577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.737563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2544.004035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2544.045356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.176394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2544.196424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2544.248610] Lustre: lustre-MDT0000: new disk, initializing [ 2544.285512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.297162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2546.061265] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2551.542444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.583041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2551.641340] Lustre: 168893:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2551.683372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2551.687331] Lustre: Skipped 1 previous similar message [ 2551.733978] Lustre: lustre-MDT0001: new disk, initializing [ 2551.776362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2551.797310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2551.802951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2553.551114] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2557.524741] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2559.102581] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2562.328643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2562.368314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2562.517586] Lustre: lustre-OST0000: new disk, initializing [ 2562.521636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2562.572779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.119898] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2566.538179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2566.544935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2566.595402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2569.133233] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2571.380217] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2572.202505] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:49:07 (1767386947) [ 2663.903910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.904249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.909584] Lustre: Skipped 2 previous similar messages [ 2663.912196] Lustre: Skipped 1 previous similar message [ 2669.024180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.029111] Lustre: Skipped 3 previous similar messages [ 2669.389894] LustreError: 172400:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2669.394122] LustreError: 172400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2669.443154] Lustre: server umount lustre-MDT0000 complete [ 2671.111998] LustreError: 170642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387048 with bad export cookie 7102464120579493138 [ 2671.114584] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.120390] LustreError: 170642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2674.143726] LustreError: 171087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.143849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.154248] LustreError: 171087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2674.162541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2676.191663] LustreError: 168899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.191794] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2681.311656] LustreError: 171087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.311679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2685.407230] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2685.468145] LustreError: 172602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2685.471843] LustreError: 172602:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2685.514902] Lustre: server umount lustre-MDT0001 complete [ 2693.355560] LustreError: 172806:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2693.359848] LustreError: 172806:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2693.411091] Lustre: server umount lustre-OST0000 complete [ 2697.475571] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2701.561064] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2706.365178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2709.518553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2712.387375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2715.266292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2718.197611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2718.229424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.328958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2718.343341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2718.396361] Lustre: lustre-MDT0000: new disk, initializing [ 2718.431671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.442755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2720.187924] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2725.611130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.650351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2725.702479] Lustre: 175141:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2725.746469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2725.750492] Lustre: Skipped 1 previous similar message [ 2725.803375] Lustre: lustre-MDT0001: new disk, initializing [ 2725.858865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2725.865052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2727.480421] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2731.464213] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2733.342300] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2736.519798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2736.551421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2736.684942] Lustre: lustre-OST0000: new disk, initializing [ 2736.688725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2736.728553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.732097] Lustre: Skipped 1 previous similar message [ 2739.204335] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2743.168108] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2744.325389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2744.331679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2744.381788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2745.005430] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2745.117815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2746.872432] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2746.985700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2749.408492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2749.410076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.411012] LustreError: Skipped 1 previous similar message [ 2749.415447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2749.418144] Lustre: Skipped 1 previous similar message [ 2753.649146] LustreError: 178081:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2753.652784] LustreError: 178081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2753.705059] Lustre: server umount lustre-OST0000 complete [ 2756.576424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2756.581832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.589288] Lustre: Skipped 1 previous similar message [ 2768.865476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.869065] Lustre: Skipped 6 previous similar messages [ 2769.375147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2769.432710] LustreError: 178282:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2769.436392] LustreError: 178282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2769.471652] Lustre: server umount lustre-MDT0000 complete [ 2771.233087] LustreError: 175924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387148 with bad export cookie 7102464120579651968 [ 2771.239550] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.243350] LustreError: 175924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2771.385513] Lustre: server umount lustre-MDT0001 complete [ 2775.304679] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:52:30 (1767387150) [ 2782.264698] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2786.481445] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2789.878888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2792.956110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.767578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2798.803377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2802.070772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2802.101859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.233720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2802.253619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2802.307979] Lustre: lustre-MDT0000: new disk, initializing [ 2802.339583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.350951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2804.068864] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2809.498404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2809.536314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2809.594801] Lustre: 181470:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2809.758749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2811.446271] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2815.408701] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2817.265186] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2820.556088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.602188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2823.433573] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2827.357153] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2829.329373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2829.393758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2839.529213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.530129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.531322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.534164] LustreError: Skipped 1 previous similar message [ 2839.541658] Lustre: Skipped 2 previous similar messages [ 2841.202104] LustreError: 184020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2841.206218] LustreError: 184020:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2841.264611] Lustre: server umount lustre-OST0000 complete [ 2848.759440] Lustre: server umount lustre-MDT0000 complete [ 2849.759933] LustreError: 181481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.765013] LustreError: 181481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2850.400944] LustreError: 182256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387227 with bad export cookie 7102464120579652836 [ 2850.406866] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.411279] LustreError: 182256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2850.571118] Lustre: server umount lustre-MDT0001 complete [ 2853.959333] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:53:49 (1767387229) [ 2860.473848] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 2864.568261] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2868.402271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2871.372113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2874.169691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2877.001383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2880.380080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2880.418179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.538421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2880.554843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2880.558599] Lustre: Skipped 3 previous similar messages [ 2880.606556] Lustre: lustre-MDT0000: new disk, initializing [ 2880.608013] Lustre: Skipped 2 previous similar messages [ 2880.644524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.648271] Lustre: Skipped 2 previous similar messages [ 2880.657980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2880.662811] Lustre: Skipped 2 previous similar messages [ 2882.213229] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2887.615095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2887.660104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2887.717333] Lustre: 187411:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2887.917829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2889.820938] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2893.890893] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2895.666953] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2898.659083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2898.704113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2901.676388] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2904.654662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2905.792403] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2909.669226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.671993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.673871] LustreError: Skipped 1 previous similar message [ 2909.681781] Lustre: Skipped 2 previous similar messages [ 2909.685443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.690615] Lustre: Skipped 5 previous similar messages [ 2910.578567] LustreError: 189957:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2910.582979] LustreError: 189957:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2910.654172] Lustre: server umount lustre-OST0000 complete [ 2913.760445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2914.686336] Lustre: server umount lustre-MDT0000 complete [ 2914.785203] LustreError: 188206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.792497] LustreError: 188206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2916.391249] LustreError: 187402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387293 with bad export cookie 7102464120579653893 [ 2916.395810] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.396514] LustreError: 187402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2916.537741] Lustre: server umount lustre-MDT0001 complete [ 2921.607573] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2923.149338] Key type lgssc unregistered [ 2923.325642] LNet: 190891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2923.331280] LNetError: 190891:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2923.342792] LNet: Removed LNI 192.168.203.105@tcp [ 2923.829722] Key type .llcrypt unregistered [ 2923.831925] Key type ._llcrypt unregistered [ 2935.688542] Key type ._llcrypt registered [ 2935.690518] Key type .llcrypt registered [ 2935.759976] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2936.079863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2936.098202] alg: No test for adler32 (adler32-zlib) [ 2936.995735] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2937.115732] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2938.727184] Key type lgssc registered [ 2939.352404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2943.198465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2943.210034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.427823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.195371] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2950.328313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2950.471736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2951.811674] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2954.380137] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2956.160267] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2959.206891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2959.389758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.000227] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2964.713778] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2968.628129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2983.911087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.914796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2983.918891] Lustre: Skipped 1 previous similar message [ 2983.919431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2983.921594] Lustre: Skipped 1 previous similar message [ 2988.467138] LustreError: 194533:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2988.534602] Lustre: server umount lustre-OST0000 complete [ 2991.584837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2991.589831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.598668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.144948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.145393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.430144] LustreError: 194735:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2996.434379] LustreError: 194735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2996.479258] Lustre: server umount lustre-MDT0000 complete [ 2998.316796] LustreError: 194434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387375 with bad export cookie 13387868794948740938 [ 2998.323454] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.325065] LustreError: 194434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2998.416515] LustreError: 194938:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2998.419272] LustreError: 194938:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2998.485199] Lustre: server umount lustre-MDT0001 complete [ 3003.187006] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3004.685074] Key type lgssc unregistered [ 3004.860757] LNet: 195470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3004.865542] LNetError: 195470:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3004.877558] LNet: Removed LNI 192.168.203.105@tcp [ 3005.319092] Key type .llcrypt unregistered [ 3005.321443] Key type ._llcrypt unregistered [ 3017.171496] Key type ._llcrypt registered [ 3017.173360] Key type .llcrypt registered [ 3017.223442] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3017.531939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3017.590966] alg: No test for adler32 (adler32-zlib) [ 3018.461190] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3018.537702] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3020.135203] Key type lgssc registered [ 3020.571074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3024.176165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3024.186445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.388111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.933761] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3030.687648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3030.872197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3032.570923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3035.047742] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3036.765400] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3039.932654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3040.124117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.698437] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3045.319232] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3050.349194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3055.586401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.590986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.593198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.599027] Lustre: Skipped 1 previous similar message [ 3060.708874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.712805] Lustre: Skipped 3 previous similar messages [ 3065.826678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.828640] Lustre: Skipped 2 previous similar messages [ 3067.359086] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3067.377118] LustreError: 199091:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3067.410234] Lustre: server umount lustre-OST0000 complete [ 3069.369052] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f5286e08000.ost_server_uuid 50 [ 3070.139637] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f5286e08000.ost_server_uuid in DISCONN state after 0 sec [ 3070.916326] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3082.208391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3082.213368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.221694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.225558] Lustre: Skipped 1 previous similar message [ 3086.305111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.305982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.354899] LustreError: 199495:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3087.356627] LustreError: 199495:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3087.379628] Lustre: server umount lustre-MDT0000 complete [ 3089.031617] LustreError: 196825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387466 with bad export cookie 12796827350450349070 [ 3089.038587] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.039224] LustreError: 196825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3089.140853] LustreError: 199698:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3089.144600] LustreError: 199698:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3089.205912] Lustre: server umount lustre-MDT0001 complete [ 3093.166549] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:57:48 (1767387468) [ 3096.292156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.485101] LustreError: 200368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.491199] LustreError: 200368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3096.510113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.224674] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3101.665103] LustreError: 200368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.391274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3102.582616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.298296] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3106.934512] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3108.580705] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3111.788192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3111.942674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.587754] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3117.111870] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3120.376611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3120.419357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3120.516898] Lustre: lustre-OST0001: new disk, initializing [ 3120.520043] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3121.030051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3121.036056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3121.086143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3122.096686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3123.073942] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3126.921323] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3128.288034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3128.290021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.293276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3128.294854] Lustre: Skipped 1 previous similar message [ 3132.386526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3132.387740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.192084] LustreError: 203573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3134.194842] LustreError: 203573:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3134.215299] Lustre: server umount lustre-OST0000 complete [ 3136.170916] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3136.249072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3137.508517] LustreError: 201964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.514629] LustreError: 201964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3137.878817] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3137.953801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3142.627252] LustreError: 202953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.636830] LustreError: 202953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3147.747924] LustreError: 201966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.758066] LustreError: 201966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3151.841436] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.841932] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.846369] LustreError: Skipped 1 previous similar message [ 3151.855217] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3151.861280] Lustre: Skipped 4 previous similar messages [ 3154.349088] LustreError: 204275:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3154.350750] LustreError: 204275:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3154.380869] Lustre: server umount lustre-OST0001 complete [ 3162.217013] Lustre: server umount lustre-MDT0000 complete [ 3163.104847] LustreError: 200368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.118965] LustreError: 200368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3163.840355] LustreError: 200349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387541 with bad export cookie 12796827350450350134 [ 3163.843155] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.845185] LustreError: 200349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3163.935800] LustreError: 204680:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3163.939066] LustreError: 204680:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3163.994366] Lustre: server umount lustre-MDT0001 complete [ 3166.268978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3167.421251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3169.030196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3170.696581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3173.659849] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:59:09 (1767387549) [ 3175.873953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.987116] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3176.000310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3176.078862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.080986] Lustre: Skipped 1 previous similar message [ 3177.630810] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3181.513719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3181.568481] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3181.592391] Lustre: 206170:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3183.305833] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3185.724170] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3187.205867] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3190.372321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3190.464360] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3191.286099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3192.740684] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3195.161903] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3197.718898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3197.761244] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3200.275296] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3202.756043] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3205.609104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.611672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.616898] Lustre: Skipped 4 previous similar messages [ 3205.623479] Lustre: Skipped 5 previous similar messages [ 3217.887119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.912161] LustreError: 209365:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3217.916312] LustreError: 209365:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3217.939710] Lustre: server umount lustre-OST0000 complete [ 3218.681557] LustreError: 207798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.204413] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.209310] LustreError: Skipped 1 previous similar message [ 3235.301141] Lustre: server umount lustre-OST0001 complete [ 3238.368411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.376261] Lustre: Skipped 2 previous similar messages [ 3243.248383] Lustre: server umount lustre-MDT0000 complete [ 3245.008497] LustreError: 206163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387622 with bad export cookie 12796827350450351520 [ 3245.012793] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3245.013593] LustreError: 206163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3245.149266] Lustre: server umount lustre-MDT0001 complete [ 3247.687076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3249.496164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3251.321443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3253.107690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3256.042319] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:00:31 (1767387631) [ 3263.137696] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3267.213714] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3271.406167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3274.507121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3277.423335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3280.389557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3283.472081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3283.508538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.638251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3283.657917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3283.706587] Lustre: lustre-MDT0000: new disk, initializing [ 3283.734237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.737864] Lustre: Skipped 3 previous similar messages [ 3283.745698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3285.462516] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3290.934173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3290.971102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3291.020620] Lustre: 213792:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3291.061548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3291.065528] Lustre: Skipped 1 previous similar message [ 3291.117850] Lustre: lustre-MDT0001: new disk, initializing [ 3291.175193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3291.180223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3292.825402] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3296.687271] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3298.420623] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3301.791851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3301.830207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3301.963852] Lustre: lustre-OST0000: new disk, initializing [ 3301.967734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3304.481152] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3306.497170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3306.499959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3306.527464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3308.333637] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3310.131232] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3310.246371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3311.988689] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3312.107799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3316.710478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3316.710699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.715452] LustreError: Skipped 2 previous similar messages [ 3316.723053] Lustre: Skipped 1 previous similar message [ 3316.726755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3316.732232] Lustre: Skipped 12 previous similar messages [ 3319.089967] LustreError: 216744:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3319.093865] LustreError: 216744:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3319.151540] Lustre: server umount lustre-OST0000 complete [ 3327.060946] Lustre: server umount lustre-MDT0000 complete [ 3328.731391] LustreError: 214583:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387706 with bad export cookie 12796827350450353011 [ 3328.738269] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3328.738866] LustreError: 214583:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3328.883168] Lustre: server umount lustre-MDT0001 complete [ 3332.836282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.051894] LustreError: 217624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.060277] LustreError: 217624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3334.869489] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3338.961270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3340.931398] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3343.481181] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.291401] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3348.457446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3348.635185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.639883] Lustre: Skipped 4 previous similar messages [ 3351.131630] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3353.659116] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3355.288126] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3358.496423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3360.276448] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3360.392366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3363.300815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.305446] LustreError: Skipped 1 previous similar message [ 3367.397767] Lustre: server umount lustre-OST0000 complete [ 3369.961492] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3371.781962] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3396.468367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.140701] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3401.818196] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3402.634465] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3402.788382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3402.795982] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3402.796033] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3410.032205] LustreError: 222021:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3410.036445] LustreError: 222021:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3410.091526] Lustre: server umount lustre-OST0000 complete [ 3426.271169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3426.355152] Lustre: server umount lustre-MDT0000 complete [ 3428.039213] LustreError: 217603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387805 with bad export cookie 12796827350450353893 [ 3428.044859] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.046907] LustreError: 217603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3428.197319] Lustre: server umount lustre-MDT0001 complete [ 3432.354922] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:03:27 (1767387807) [ 3435.462731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.706646] LustreError: 223101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.718561] LustreError: 223101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3437.321844] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3441.386388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3443.369789] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3445.886684] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3447.544916] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3450.765193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3453.468207] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3456.004740] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3457.863751] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3462.114365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3463.827433] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3463.945808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3467.239560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.280054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3467.385994] Lustre: lustre-OST0001: new disk, initializing [ 3467.391643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3470.016294] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3474.159947] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3475.817783] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3476.506057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3476.512555] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3476.557766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3476.967911] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3478.741564] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3478.860343] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3481.573606] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.576751] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.578550] LustreError: Skipped 3 previous similar messages [ 3481.588665] Lustre: Skipped 8 previous similar messages [ 3481.592581] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3481.598855] Lustre: Skipped 15 previous similar messages [ 3485.836459] Lustre: server umount lustre-OST0001 complete [ 3488.560740] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3488.679569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3490.520943] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3490.638836] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3515.357108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3515.462713] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.466434] Lustre: Skipped 5 previous similar messages [ 3518.042939] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3520.490920] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3520.496302] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3520.496358] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3520.504458] Lustre: Skipped 2 previous similar messages [ 3520.738940] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3521.569356] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3528.577742] Lustre: server umount lustre-OST0001 complete [ 3547.378156] LustreError: 229220:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3547.382276] LustreError: 229220:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3547.437565] Lustre: server umount lustre-OST0000 complete [ 3555.502718] Lustre: server umount lustre-MDT0000 complete [ 3557.314224] LustreError: 223082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387934 with bad export cookie 12796827350450354852 [ 3557.319875] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.321950] LustreError: 223082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3563.536288] Lustre: server umount lustre-MDT0001 complete [ 3566.211444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.073969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3569.916931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3571.822342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3575.179237] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:05:50 (1767387950) [ 3578.371677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.500196] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3578.515442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3580.261354] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3584.310415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3584.379731] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3584.397997] Lustre: 231114:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3586.170831] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3588.688148] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3590.339272] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3593.536710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3593.636674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3596.130396] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3598.788337] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3609.025047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3609.064890] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3611.798064] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3614.350680] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3617.283936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3619.097810] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3619.198928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3620.965968] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3621.057308] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3622.773856] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f528550e800.ost_server_uuid 50 [ 3623.291896] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f528550e800.ost_server_uuid in FULL state after 0 sec [ 3623.759679] Lustre: Permanently deactivating lustre-OST0001 [ 3623.768709] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3624.801313] Lustre: Permanently reactivating lustre-OST0001 [ 3630.565039] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3630.569440] LustreError: Skipped 4 previous similar messages [ 3632.200317] Lustre: server umount lustre-OST0001 complete [ 3635.687870] LustreError: 232737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.697859] LustreError: 232737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 30 previous similar messages [ 3640.101600] Lustre: server umount lustre-OST0000 complete [ 3647.860683] Lustre: server umount lustre-MDT0000 complete [ 3649.668199] LustreError: 232753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388027 with bad export cookie 12796827350450356126 [ 3649.673726] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.675431] LustreError: 232753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3649.817308] Lustre: server umount lustre-MDT0001 complete [ 3652.423102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.220087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3655.784922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3657.596254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.816871] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:07:16 (1767388036) [ 3661.325797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3664.578067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.697811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3664.715422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3664.723473] Lustre: Skipped 5 previous similar messages [ 3666.527833] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3670.441664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3670.493517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3670.507250] Lustre: 237258:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3672.296760] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3674.834546] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3676.480724] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3679.646721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3679.687389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3679.808290] Lustre: Permanently deactivating lustre-OST0000 [ 3679.816920] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3682.432769] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3684.906630] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3687.935771] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3688.158455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3690.743347] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3693.227789] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3697.732223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3699.549161] Lustre: Permanently reactivating lustre-OST0000 [ 3699.559126] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3699.563601] Lustre: Skipped 2 previous similar messages [ 3706.357708] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3706.387964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3712.765750] Lustre: server umount lustre-OST0001 complete [ 3720.358603] Lustre: server umount lustre-OST0000 complete [ 3728.180636] Lustre: server umount lustre-MDT0000 complete [ 3729.963841] LustreError: 237250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388107 with bad export cookie 12796827350450357757 [ 3729.970790] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.974855] LustreError: 237250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3730.138135] Lustre: server umount lustre-MDT0001 complete [ 3734.579281] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3736.028984] Key type lgssc unregistered [ 3736.189698] LNet: 241649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3736.194981] LNetError: 241649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3736.207550] LNet: Removed LNI 192.168.203.105@tcp [ 3736.664762] Key type .llcrypt unregistered [ 3736.666995] Key type ._llcrypt unregistered [ 3747.515914] Key type ._llcrypt registered [ 3747.517580] Key type .llcrypt registered [ 3747.553814] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3747.838932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.845069] alg: No test for adler32 (adler32-zlib) [ 3748.720373] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3748.811066] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3750.399148] Key type lgssc registered [ 3750.927253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3752.511351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3755.625680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3755.637043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.866983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.557441] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3762.647883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3762.686887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3762.739500] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3762.751940] Lustre: 243064:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3762.758972] Lustre: Permanently deactivating lustre-MDT0001 [ 3762.766768] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3762.902686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.524500] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3765.368296] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3765.373678] LustreError: 243107:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3765.381698] LustreError: 243107:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3765.388550] LustreError: 243107:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3767.175358] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3768.896331] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3772.204890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3772.362429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.413553] LustreError: 244692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3773.424153] LustreError: 244692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3774.839073] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3777.232476] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3778.534416] LustreError: 244835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.489147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3780.614190] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3781.684725] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3781.685656] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3781.691687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3783.230781] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3785.764667] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3787.490868] Lustre: Permanently reactivating lustre-MDT0001 [ 3787.501108] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3787.505770] Lustre: Skipped 2 previous similar messages [ 3790.980106] Lustre: lustre-MDT0001: Client 4343bd25-faa2-416b-8b91-ea3cc7f622b5 (at 192.168.203.5@tcp) reconnecting [ 3793.337347] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3794.001829] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3796.002431] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3796.121348] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3797.796311] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3797.911420] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3798.964340] Lustre: Permanently deactivating lustre-MDT0001 [ 3798.974747] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3798.979039] Lustre: Skipped 2 previous similar messages [ 3800.218269] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3802.083597] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.084748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.087272] Lustre: Skipped 3 previous similar messages [ 3807.202579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3807.206554] Lustre: Skipped 6 previous similar messages [ 3808.208640] LustreError: 247145:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3808.251478] Lustre: server umount lustre-MDT0000 complete [ 3809.942705] LustreError: 243057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388187 with bad export cookie 3296987386859737043 [ 3809.948723] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.949551] LustreError: 243057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3810.047616] LustreError: 247346:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3810.050178] LustreError: 247346:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3810.104233] Lustre: server umount lustre-MDT0001 complete [ 3817.714165] LustreError: 247546:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3817.718336] LustreError: 247546:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3817.740269] Lustre: server umount lustre-OST0000 complete [ 3822.895149] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:09:58 (1767388198) [ 3827.775276] LustreError: 248552:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3827.779219] LustreError: 248552:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3827.841156] Lustre: server umount lustre-OST0001 complete [ 3830.529094] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3834.503858] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3838.608730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3841.751835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3844.659073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3847.797524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.079358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.124144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.234239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3851.236835] Lustre: Skipped 2 previous similar messages [ 3851.247559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3851.281127] Lustre: lustre-MDT0000: new disk, initializing [ 3851.308631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.316092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3853.006614] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3858.456718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3858.495612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3858.550416] Lustre: 250733:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3858.591272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3858.595108] Lustre: Skipped 1 previous similar message [ 3858.642894] Lustre: lustre-MDT0001: new disk, initializing [ 3858.697585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3858.703133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3860.339477] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3864.384996] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3866.001497] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3869.252470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3869.291898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3869.431066] Lustre: lustre-OST0000: new disk, initializing [ 3869.435477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3869.506132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.509852] Lustre: Skipped 1 previous similar message [ 3872.070383] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3874.800116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3874.806836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.828398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3876.135960] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.100815] LustreError: 250739:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3882.298890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3882.340798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3882.422818] Lustre: lustre-OST0001: new disk, initializing [ 3882.425810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3882.586836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3882.590671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3882.623309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3885.039836] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3887.192103] LustreError: 250739:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3887.207378] LustreError: 250739:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3889.103651] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3889.870943] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3897.303133] LustreError: 250739:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3898.342576] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3898.342953] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3898.346407] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3898.347471] LustreError: Skipped 1 previous similar message [ 3898.355173] Lustre: Skipped 1 previous similar message [ 3903.464739] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3903.468806] Lustre: Skipped 3 previous similar messages [ 3904.306456] LustreError: 254233:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3904.310700] LustreError: 254233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3904.338624] Lustre: server umount lustre-OST0001 complete [ 3904.763620] LustreError: 252464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.582436] LustreError: 254234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.591770] LustreError: 254234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3913.701294] LustreError: 254234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.711613] LustreError: 254234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3918.823870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.829392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3918.831761] Lustre: Skipped 1 previous similar message [ 3923.124187] LustreError: 254460:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 3923.128072] LustreError: 254460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3923.187753] Lustre: server umount lustre-OST0000 complete [ 3925.472722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3925.479471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.058431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3929.062438] Lustre: Skipped 3 previous similar messages [ 3931.247522] Lustre: server umount lustre-MDT0000 complete [ 3933.008105] LustreError: 251515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388310 with bad export cookie 3296987386859739626 [ 3933.014492] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3933.015474] LustreError: 251515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3939.155314] LustreError: 254863:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3939.159424] LustreError: 254863:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3939.254873] Lustre: server umount lustre-MDT0001 complete [ 3944.503049] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3946.091438] Key type lgssc unregistered [ 3946.260581] LNet: 255395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3946.265643] LNetError: 255395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3946.277722] LNet: Removed LNI 192.168.203.105@tcp [ 3946.742915] Key type .llcrypt unregistered [ 3946.745203] Key type ._llcrypt unregistered [ 3962.342476] Key type ._llcrypt registered [ 3962.344240] Key type .llcrypt registered [ 3962.397838] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3970.421232] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3970.734818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.776025] alg: No test for adler32 (adler32-zlib) [ 3971.637283] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3971.734855] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3973.335167] Key type lgssc registered [ 3973.881872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3978.029792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3981.212742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3984.122852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.042050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.059574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.079966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3990.085306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.183481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3991.201189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3991.251889] Lustre: lustre-MDT0000: new disk, initializing [ 3991.280507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.289761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3992.933721] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3998.231734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3998.271962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3998.329894] Lustre: 259320:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3998.390765] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3998.393923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3998.397957] Lustre: Skipped 1 previous similar message [ 3998.445601] Lustre: lustre-MDT0001: new disk, initializing [ 3998.494922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3998.516707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3998.523352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4000.268894] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4004.160857] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4005.307441] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4008.055042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.096090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4008.233411] Lustre: lustre-OST0000: new disk, initializing [ 4008.236414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4008.282343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4010.562596] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4013.070174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4013.076638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4013.124969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4014.404255] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4015.633684] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4015.699731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4017.389493] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4017.494169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4023.268020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4023.268768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.273479] LustreError: Skipped 1 previous similar message [ 4023.278747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4023.282679] Lustre: Skipped 1 previous similar message [ 4024.434172] LustreError: 262260:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4024.493991] Lustre: server umount lustre-OST0000 complete [ 4028.386753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.387838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4028.396652] Lustre: Skipped 2 previous similar messages [ 4028.402623] Lustre: Skipped 1 previous similar message [ 4032.405704] LustreError: 262463:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4032.409263] LustreError: 262463:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4032.446657] Lustre: server umount lustre-MDT0000 complete [ 4034.146988] LustreError: 261066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388411 with bad export cookie 11556676733692124795 [ 4034.153435] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4034.154967] LustreError: 261066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4034.250922] LustreError: 262665:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4034.254666] LustreError: 262665:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4034.314762] Lustre: server umount lustre-MDT0001 complete [ 4038.494699] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 16:13:34 (1767388414) [ 4041.654553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.863098] LustreError: 263336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.873853] LustreError: 263336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4041.902059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.542374] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4047.330563] LustreError: 263336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.677496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.841791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.235882] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4051.824078] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4053.511243] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4056.576504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4056.671484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.726893] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4061.224051] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4113.382335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.387488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.388829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.390346] Lustre: Skipped 1 previous similar message [ 4116.723127] LustreError: 265680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4116.727365] LustreError: 265680:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4116.806990] Lustre: server umount lustre-OST0000 complete [ 4118.881544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4124.726718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4124.904193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.401344] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4129.905897] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4133.102581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.110057] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.110302] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4133.120360] Lustre: Skipped 1 previous similar message [ 4169.190281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.191879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.195411] LustreError: Skipped 1 previous similar message [ 4169.196161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.196170] Lustre: Skipped 1 previous similar message [ 4169.203242] Lustre: Skipped 1 previous similar message [ 4170.480208] LustreError: 267345:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4170.484390] LustreError: 267345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4170.562796] Lustre: server umount lustre-OST0000 complete [ 4174.307912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.316224] Lustre: Skipped 1 previous similar message [ 4178.575606] LustreError: 267546:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4178.579835] LustreError: 267546:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4178.624678] Lustre: server umount lustre-MDT0000 complete [ 4179.425436] LustreError: 263335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.435223] LustreError: 263335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4180.460449] LustreError: 263316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388557 with bad export cookie 11556676733692125677 [ 4180.466575] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.467341] LustreError: 263316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4180.639853] Lustre: server umount lustre-MDT0001 complete [ 4185.034081] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4186.555064] Key type lgssc unregistered [ 4186.727669] LNet: 268278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4186.732789] LNetError: 268278:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4186.743595] LNet: Removed LNI 192.168.203.105@tcp [ 4187.233285] Key type .llcrypt unregistered [ 4187.235934] Key type ._llcrypt unregistered [ 4198.899703] Key type ._llcrypt registered [ 4198.901470] Key type .llcrypt registered [ 4198.968646] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4199.415490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4199.448945] alg: No test for adler32 (adler32-zlib) [ 4200.372737] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4200.495037] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4202.095158] Key type lgssc registered [ 4202.676363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4206.619047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4206.630699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.836898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.683407] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4213.878160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4214.084565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4216.010774] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4218.655580] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4220.449998] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4223.797372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4223.980395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.691995] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4229.284408] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4233.202260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4245.472320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4245.477328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.485750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4248.545945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4248.549267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4248.550891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.993154] LustreError: 272673:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4251.039079] Lustre: server umount lustre-OST0000 complete [ 4253.666207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.667533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.674129] Lustre: Skipped 1 previous similar message [ 4253.680703] Lustre: Skipped 2 previous similar messages [ 4258.784893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.788804] Lustre: Skipped 1 previous similar message [ 4258.957192] LustreError: 272874:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4258.961467] LustreError: 272874:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4258.995086] Lustre: server umount lustre-MDT0000 complete [ 4260.773803] LustreError: 269634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388638 with bad export cookie 2597377593067989817 [ 4260.778211] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.781515] LustreError: 269634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4260.885336] LustreError: 273079:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4260.888232] LustreError: 273079:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4260.943841] Lustre: server umount lustre-MDT0001 complete [ 4266.003261] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4267.584561] Key type lgssc unregistered [ 4267.764478] LNet: 273610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4267.769547] LNetError: 273610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4267.782620] LNet: Removed LNI 192.168.203.105@tcp [ 4268.131339] Key type .llcrypt unregistered [ 4268.132414] Key type ._llcrypt unregistered [ 4276.894054] Key type ._llcrypt registered [ 4276.896151] Key type .llcrypt registered [ 4276.954527] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4277.437440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.466266] alg: No test for adler32 (adler32-zlib) [ 4278.389809] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4278.512704] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4280.127164] Key type lgssc registered [ 4280.739055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4284.426375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.438767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.673564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.415716] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4291.546071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4291.748948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4293.535406] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4296.229611] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4298.039092] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4301.330580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4301.502804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.091377] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4306.654167] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4310.765796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4318.176347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.181253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.189604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.253087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4321.253706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.254134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.270050] Lustre: Skipped 1 previous similar message [ 4322.737143] LustreError: 277182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4322.795324] Lustre: server umount lustre-OST0000 complete [ 4326.370428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.371430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.378262] Lustre: Skipped 1 previous similar message [ 4326.384808] Lustre: Skipped 1 previous similar message [ 4330.764813] LustreError: 277384:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4330.768940] LustreError: 277384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4330.803213] Lustre: server umount lustre-MDT0000 complete [ 4331.488871] LustreError: 275514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.498681] LustreError: 275514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4332.584831] LustreError: 274770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388710 with bad export cookie 16766355045291465136 [ 4332.590987] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.592276] LustreError: 274770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4332.695356] LustreError: 277587:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4332.698381] LustreError: 277587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4332.745690] Lustre: server umount lustre-MDT0001 complete [ 4337.690983] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4339.219149] Key type lgssc unregistered [ 4339.399790] LNet: 278119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4339.404819] LNetError: 278119:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4339.417779] LNet: Removed LNI 192.168.203.105@tcp [ 4339.905147] Key type .llcrypt unregistered [ 4339.907283] Key type ._llcrypt unregistered [ 4352.094200] Key type ._llcrypt registered [ 4352.095983] Key type .llcrypt registered [ 4352.162804] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4352.578687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4352.613385] alg: No test for adler32 (adler32-zlib) [ 4353.530892] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4353.654888] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4355.271151] Key type lgssc registered [ 4355.866189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4359.851515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4359.862677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.081621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.848332] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4366.867396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4367.064724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4368.721712] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4371.345623] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4373.056040] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4376.266647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4376.431959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.668338] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4380.863421] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4385.645859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4398.049241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.054242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.063121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4401.124385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.125464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4401.127189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.141103] Lustre: Skipped 1 previous similar message [ 4402.546178] LustreError: 282509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4402.601291] Lustre: server umount lustre-OST0000 complete [ 4406.242643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.243471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4406.250549] Lustre: Skipped 1 previous similar message [ 4406.257044] Lustre: Skipped 1 previous similar message [ 4410.512282] LustreError: 282712:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4410.516712] LustreError: 282712:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4410.550855] Lustre: server umount lustre-MDT0000 complete [ 4411.360878] LustreError: 279494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.370339] LustreError: 279494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4412.261957] LustreError: 281112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388789 with bad export cookie 5844510342307884484 [ 4412.267074] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4412.269594] LustreError: 281112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4412.363096] LustreError: 282915:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4412.365707] LustreError: 282915:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4412.411752] Lustre: server umount lustre-MDT0001 complete [ 4417.221710] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4418.699181] Key type lgssc unregistered [ 4418.878677] LNet: 283446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.883486] LNetError: 283446:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4418.894574] LNet: Removed LNI 192.168.203.105@tcp [ 4419.333259] Key type .llcrypt unregistered [ 4419.335608] Key type ._llcrypt unregistered [ 4428.330162] Key type ._llcrypt registered [ 4428.331632] Key type .llcrypt registered [ 4428.373985] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4428.772410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4428.806448] alg: No test for adler32 (adler32-zlib) [ 4429.647478] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4429.730689] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4431.303189] Key type lgssc registered [ 4431.679345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4435.024799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4435.036468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.245453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.000280] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4441.997114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4442.180629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4443.833325] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4446.486918] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4448.307321] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4451.485439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4451.650944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.075587] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4456.732293] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4460.907652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4468.192270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.197245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.205610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.269151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.269831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.281334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4472.690148] LustreError: 287054:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4472.745794] Lustre: server umount lustre-OST0000 complete [ 4476.386057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.395266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.399218] Lustre: Skipped 2 previous similar messages [ 4480.653268] LustreError: 287256:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4480.657495] LustreError: 287256:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4480.692641] Lustre: server umount lustre-MDT0000 complete [ 4481.504824] LustreError: 284653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.514914] LustreError: 284653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4482.463351] LustreError: 284608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388859 with bad export cookie 12275816596690161426 [ 4482.468492] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.470839] LustreError: 284608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4482.574178] LustreError: 287458:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4482.577103] LustreError: 287458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4482.642204] Lustre: server umount lustre-MDT0001 complete [ 4487.721585] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4489.216372] Key type lgssc unregistered [ 4489.378757] LNet: 287990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4489.383763] LNetError: 287990:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4489.397679] LNet: Removed LNI 192.168.203.105@tcp [ 4489.843929] Key type .llcrypt unregistered [ 4489.846161] Key type ._llcrypt unregistered [ 4494.675925] Key type ._llcrypt registered [ 4494.678014] Key type .llcrypt registered [ 4494.742686] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4502.169503] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 4509.994557] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4510.348123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4510.381635] alg: No test for adler32 (adler32-zlib) [ 4511.240154] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4511.341658] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4512.927158] Key type lgssc registered [ 4513.506344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4517.928985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.118054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4524.099805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4527.121451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4530.358927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4530.387310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4530.396587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.510315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4531.530274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4531.584593] Lustre: lustre-MDT0000: new disk, initializing [ 4531.616594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.623315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4533.366149] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4538.742097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4538.780831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4538.834346] Lustre: 291474:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4538.876292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4538.880222] Lustre: Skipped 1 previous similar message [ 4538.927511] Lustre: lustre-MDT0001: new disk, initializing [ 4538.973945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4538.995616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4539.002411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4540.630213] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4544.594388] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4546.396655] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4549.466353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4549.515222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4549.655755] Lustre: lustre-OST0000: new disk, initializing [ 4549.658987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4549.701888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.239023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4550.242551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4550.282133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4552.287850] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4556.274699] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4557.958225] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4558.036348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4559.760649] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4559.866551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4560.868093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.869364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4560.869764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4560.875956] Lustre: Skipped 1 previous similar message [ 4565.988671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4565.992625] Lustre: Skipped 2 previous similar messages [ 4571.109807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.113814] Lustre: Skipped 1 previous similar message [ 4575.199172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4575.227202] LustreError: 294414:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4575.286222] Lustre: server umount lustre-OST0000 complete [ 4579.808596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4579.813430] LustreError: Skipped 1 previous similar message [ 4579.816229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.824647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.346461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.184484] LustreError: 294617:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4583.189037] LustreError: 294617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4583.228320] Lustre: server umount lustre-MDT0000 complete [ 4584.903086] LustreError: 292256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388962 with bad export cookie 8765328131505642705 [ 4584.909398] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4584.910673] LustreError: 292256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4585.001519] LustreError: 294820:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4585.004920] LustreError: 294820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4585.069547] Lustre: server umount lustre-MDT0001 complete [ 4588.760805] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:22:44 (1767388964) [ 4591.873041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.108730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.655392] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4597.217925] LustreError: 295490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.227299] LustreError: 295490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4597.417237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4597.596510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4599.415923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4602.007213] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4603.761927] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4606.993832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4607.170573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.760888] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4612.325572] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4622.819558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.820776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.822248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.822258] Lustre: Skipped 2 previous similar messages [ 4622.828795] Lustre: Skipped 1 previous similar message [ 4627.119948] LustreError: 297748:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4627.124166] LustreError: 297748:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4627.180301] Lustre: server umount lustre-OST0000 complete [ 4631.009526] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4631.013340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.022139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.026039] Lustre: Skipped 1 previous similar message [ 4643.295202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4643.347631] LustreError: 297950:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4643.352353] LustreError: 297950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4643.380829] Lustre: server umount lustre-MDT0000 complete [ 4645.119934] LustreError: 295469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389022 with bad export cookie 8765328131505643587 [ 4645.123819] LustreError: 296211:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4645.126104] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4645.130209] LustreError: 295469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4645.283904] Lustre: server umount lustre-MDT0001 complete [ 4649.956423] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4651.444080] Key type lgssc unregistered [ 4651.612666] LNet: 298686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4651.618238] LNetError: 298686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4651.628650] LNet: Removed LNI 192.168.203.105@tcp [ 4652.099579] Key type .llcrypt unregistered [ 4652.101970] Key type ._llcrypt unregistered [ 4663.642737] Key type ._llcrypt registered [ 4663.644526] Key type .llcrypt registered [ 4663.706120] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 4670.141526] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4670.472454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4670.504656] alg: No test for adler32 (adler32-zlib) [ 4671.391566] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4671.506589] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4673.095171] Key type lgssc registered [ 4673.704831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4677.921147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4680.779373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4683.154161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.769268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4688.035872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4690.611519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4693.947644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4693.981737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4693.990908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.123486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4695.143664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4695.200644] Lustre: lustre-MDT0000: new disk, initializing [ 4695.234060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.243965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4697.178834] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4702.776222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4702.822202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4702.883572] Lustre: 302462:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4702.930089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4702.934158] Lustre: Skipped 1 previous similar message [ 4702.985095] Lustre: lustre-MDT0001: new disk, initializing [ 4703.043993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4703.066213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4703.073645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4704.930574] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4709.015253] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4710.463376] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4713.626344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4713.653121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4713.748898] Lustre: lustre-OST2710: new disk, initializing [ 4713.750825] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4713.779978] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4716.371794] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4720.352498] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4721.166492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4721.173466] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4721.232792] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4723.712978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4723.758777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4723.855157] Lustre: lustre-OST03e8: new disk, initializing [ 4723.859227] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4723.920292] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4726.091936] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4728.354816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4728.361844] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4728.419458] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4728.566179] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4731.762237] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4731.880162] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4733.464104] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4733.573580] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4735.370259] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4735.493144] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4737.142475] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4737.260254] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4739.040645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4739.046504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4739.055543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.651864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.652318] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.663856] Lustre: Skipped 2 previous similar messages [ 4744.784945] LustreError: 306803:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4744.826230] Lustre: server umount lustre-MDT0000 complete [ 4746.572741] LustreError: 303243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389124 with bad export cookie 6136709856640573349 [ 4746.580233] LustreError: 303243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4746.580497] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.683092] LustreError: 307005:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4746.686615] LustreError: 307005:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4746.748012] Lustre: server umount lustre-MDT0001 complete [ 4758.968129] LustreError: 307206:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4758.972327] LustreError: 307206:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4758.995542] Lustre: server umount lustre-OST2710 complete [ 4771.000462] LustreError: 307407:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4771.004603] LustreError: 307407:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4771.071278] Lustre: server umount lustre-OST03e8 complete [ 4779.319265] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 4783.415793] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4787.700427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4790.830486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4793.793620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4796.885276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.093321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.129942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.255936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4800.275937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4800.329090] Lustre: lustre-MDT0000: new disk, initializing [ 4800.357682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.366306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4802.050342] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4807.424595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4807.465076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4807.519065] Lustre: 310201:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4807.667436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4809.271859] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4813.235182] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4814.875957] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4818.078184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4818.115923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4818.258475] Lustre: lustre-OST0000: new disk, initializing [ 4818.261545] Lustre: Skipped 1 previous similar message [ 4818.265170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4818.269041] Lustre: Skipped 2 previous similar messages [ 4818.314675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4818.318737] Lustre: Skipped 1 previous similar message [ 4820.853942] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4824.806563] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4825.610646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4825.616273] Lustre: Skipped 1 previous similar message [ 4825.620257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4825.667937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4826.716281] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4826.836674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4828.510279] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4828.620154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4830.689790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4830.690783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.694319] LustreError: Skipped 1 previous similar message [ 4830.703285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4830.708274] Lustre: Skipped 4 previous similar messages [ 4831.536144] LustreError: 313139:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4831.540369] LustreError: 313139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4831.599365] Lustre: server umount lustre-OST0000 complete [ 4835.810626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.811395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4835.818597] Lustre: Skipped 2 previous similar messages [ 4835.825707] Lustre: Skipped 1 previous similar message [ 4839.536539] Lustre: server umount lustre-MDT0000 complete [ 4841.272122] LustreError: 310982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389218 with bad export cookie 6136709856640575015 [ 4841.277209] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4841.279671] LustreError: 310982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4841.378479] LustreError: 313543:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4841.382158] LustreError: 313543:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4841.432291] Lustre: server umount lustre-MDT0001 complete [ 4845.546620] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:27:01 (1767389221) [ 4846.292643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4847.122466] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:27:02 (1767389222) [ 4848.159189] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4851.319848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4852.796591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4854.481192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.349338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.902822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.170586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.300166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4860.315178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4860.413586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.021645] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4866.118665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4866.171300] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4866.185913] Lustre: 315671:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4867.953119] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4870.419086] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4872.002570] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4875.265084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.308254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4875.412102] Lustre: Denying initial registration attempt from nid 192.168.203.105@tcp, specified as failover [ 4875.418286] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4875.424450] LustreError: 317287:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -99 [ 4875.429428] LustreError: 317287:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4875.433816] LustreError: 317287:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4875.470457] Lustre: server umount lustre-OST0000 complete [ 4875.473446] LustreError: 317287:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4879.330272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.331179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.337376] Lustre: Skipped 1 previous similar message [ 4884.680124] LustreError: 317655:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4884.683681] LustreError: 317655:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4884.716140] Lustre: server umount lustre-MDT0000 complete [ 4886.472918] LustreError: 316397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389263 with bad export cookie 6136709856640575883 [ 4886.479218] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4886.480384] LustreError: 316397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4886.635626] Lustre: server umount lustre-MDT0001 complete [ 4891.111815] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4892.476932] Key type lgssc unregistered [ 4892.644640] LNet: 318389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4892.649655] LNetError: 318389:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4892.660601] LNet: Removed LNI 192.168.203.105@tcp [ 4893.117307] Key type .llcrypt unregistered [ 4893.119665] Key type ._llcrypt unregistered [ 4897.744748] Key type ._llcrypt registered [ 4897.746897] Key type .llcrypt registered [ 4897.810836] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4898.108166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4898.113878] alg: No test for adler32 (adler32-zlib) [ 4898.975767] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4899.067365] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4900.647154] Key type lgssc registered [ 4901.218121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4904.557919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4906.340708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4908.125171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4909.947446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4910.461661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4918.827928] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4922.567869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4922.579391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.714699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4923.727097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4923.825243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4925.540791] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4929.729324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4929.784562] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4929.801473] Lustre: 321120:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4929.954871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4931.794789] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4934.499366] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4936.317699] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4939.630648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4939.674908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4939.785094] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4939.894951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.534167] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4945.211832] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4946.404708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4946.405168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.410495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4946.417506] Lustre: Skipped 1 previous similar message [ 4946.424035] Lustre: Skipped 1 previous similar message [ 4951.524546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4951.528465] Lustre: Skipped 1 previous similar message [ 4952.307193] LustreError: 323404:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4952.365409] Lustre: server umount lustre-OST0000 complete [ 4955.616770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4955.621902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.630540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4956.642732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.333960] LustreError: 323606:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4960.338231] LustreError: 323606:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4960.375372] Lustre: server umount lustre-MDT0000 complete [ 4962.170879] LustreError: 321113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389339 with bad export cookie 2384135514866231156 [ 4962.177307] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4962.178243] LustreError: 321113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4962.272788] LustreError: 323809:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4962.276298] LustreError: 323809:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4962.338728] Lustre: server umount lustre-MDT0001 complete [ 4967.231247] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4968.669609] Key type lgssc unregistered [ 4968.836709] LNet: 324340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4968.841755] LNetError: 324340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4968.852596] LNet: Removed LNI 192.168.203.105@tcp [ 4969.290709] Key type .llcrypt unregistered [ 4969.292790] Key type ._llcrypt unregistered [ 4981.329058] Key type ._llcrypt registered [ 4981.330137] Key type .llcrypt registered [ 4981.366188] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4981.695975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4981.710538] alg: No test for adler32 (adler32-zlib) [ 4982.572859] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4982.677275] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4984.279164] Key type lgssc registered [ 4984.809448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4988.389684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4988.401340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.595570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.264466] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4995.370866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4995.561781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4997.392054] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4999.916967] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5001.544327] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5004.779673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5004.952996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5007.479815] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5010.030216] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5025.765040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.767699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5025.772924] Lustre: Skipped 2 previous similar messages [ 5025.779800] Lustre: Skipped 2 previous similar messages [ 5026.565869] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 5030.413075] LustreError: 327983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5030.446969] Lustre: server umount lustre-MDT0000 complete [ 5030.885799] LustreError: 325716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5030.896274] LustreError: 325716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5031.672909] LustreError: 325710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.195992] LustreError: 325697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389409 with bad export cookie 12595338244618666658 [ 5032.202266] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5032.203496] LustreError: 325697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.000633] LustreError: 325710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.001081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.018850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5038.048406] LustreError: 326385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.049095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5038.058778] LustreError: 326385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5038.062738] Lustre: Skipped 1 previous similar message [ 5038.348929] LustreError: 328187:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5038.353278] LustreError: 328187:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5038.423593] Lustre: server umount lustre-MDT0001 complete [ 5041.472458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5045.473663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.504362] LustreError: 325026:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008da42c9d x1853242065575552/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5057.691305] LustreError: 327322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.744604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.774435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5059.175890] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5063.384498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5063.599653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5065.354462] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5066.671072] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5067.932511] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5068.722518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5070.391830] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5078.382968] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5082.085220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.092941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5082.093369] Lustre: Skipped 1 previous similar message [ 5092.324318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.328196] Lustre: Skipped 4 previous similar messages [ 5094.367224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5094.395917] LustreError: 330494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5094.400032] LustreError: 330494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5094.458962] Lustre: server umount lustre-OST0000 complete [ 5097.442362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.450347] Lustre: Skipped 1 previous similar message [ 5102.349480] LustreError: 330696:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5102.353785] LustreError: 330696:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5102.386537] Lustre: server umount lustre-MDT0000 complete [ 5102.560857] LustreError: 329001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.571847] LustreError: 329001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5104.149954] LustreError: 327334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389481 with bad export cookie 12595338244618673812 [ 5104.155915] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5104.157569] LustreError: 327334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5104.307231] Lustre: server umount lustre-MDT0001 complete [ 5109.037660] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5110.432581] Key type lgssc unregistered [ 5110.600527] LNet: 331430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5110.605355] LNetError: 331430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5110.622558] LNet: Removed LNI 192.168.203.105@tcp [ 5111.076809] Key type .llcrypt unregistered [ 5111.079077] Key type ._llcrypt unregistered [ 5123.223836] Key type ._llcrypt registered [ 5123.225880] Key type .llcrypt registered [ 5123.288067] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5123.584180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5123.624029] alg: No test for adler32 (adler32-zlib) [ 5124.499318] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5124.600059] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5126.183153] Key type lgssc registered [ 5126.752949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5130.516593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5130.528359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.728614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.449694] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5137.473069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5137.643333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5139.344173] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5142.029299] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5143.843057] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5147.106261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.107162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.114152] Lustre: Skipped 1 previous similar message [ 5148.129369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5148.133153] Lustre: Skipped 2 previous similar messages [ 5151.435909] LustreError: 334209:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5151.471270] Lustre: server umount lustre-MDT0000 complete [ 5153.229056] LustreError: 332801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389530 with bad export cookie 10864192451324502494 [ 5153.234738] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.236505] LustreError: 332801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.341618] LustreError: 334413:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5153.344680] LustreError: 334413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5153.399102] Lustre: server umount lustre-MDT0001 complete [ 5160.244595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.394425] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5160.415574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5160.534777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.164454] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5166.297466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5166.350183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5166.361058] Lustre: 335184:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5166.461849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5168.182085] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5170.722974] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5172.544453] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5176.241996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5176.352842] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5176.359911] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5176.365699] LustreError: 336842:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -2 [ 5176.370613] LustreError: 336842:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 5176.374865] LustreError: 336842:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5176.412138] Lustre: server umount lustre-OST0000 complete [ 5176.414919] LustreError: 336842:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5180.469497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5180.574867] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5180.684807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.357935] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5186.023894] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5189.168392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5189.816404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5189.859274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5189.898783] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5189.944370] Lustre: lustre-OST0001: new disk, initializing [ 5189.947947] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5189.991636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5192.565299] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5194.776352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5194.782541] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5194.827473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5195.132458] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5199.845118] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.845312] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.850100] LustreError: Skipped 1 previous similar message [ 5199.859238] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5199.864702] Lustre: Skipped 1 previous similar message [ 5202.609767] LustreError: 338964:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5202.613853] LustreError: 338964:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5202.644664] Lustre: server umount lustre-OST0001 complete [ 5204.449385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.450239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.454412] LustreError: Skipped 1 previous similar message [ 5204.462502] Lustre: Skipped 1 previous similar message [ 5204.466184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.472123] Lustre: Skipped 1 previous similar message [ 5204.962777] LustreError: 338786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5204.972842] LustreError: 338786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5210.083624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5210.087547] Lustre: Skipped 1 previous similar message [ 5210.087725] LustreError: 338786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.100286] LustreError: 338786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5210.480129] LustreError: 339165:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5210.484239] LustreError: 339165:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5210.540902] Lustre: server umount lustre-OST0000 complete [ 5212.640330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5212.645347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.513487] LustreError: 339367:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5218.517600] LustreError: 339367:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5218.553035] Lustre: server umount lustre-MDT0000 complete [ 5220.288939] LustreError: 335175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389597 with bad export cookie 10864192451324503040 [ 5220.295090] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5220.296467] LustreError: 335175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5220.463993] Lustre: server umount lustre-MDT0001 complete [ 5224.889984] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5226.365255] Key type lgssc unregistered [ 5226.530694] LNet: 340100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5226.535658] LNetError: 340100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5226.545597] LNet: Removed LNI 192.168.203.105@tcp [ 5226.977290] Key type .llcrypt unregistered [ 5226.979503] Key type ._llcrypt unregistered [ 5242.170380] Key type ._llcrypt registered [ 5242.171868] Key type .llcrypt registered [ 5242.224727] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5250.489474] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5250.816707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5250.866676] alg: No test for adler32 (adler32-zlib) [ 5251.760738] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5251.884063] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5253.495187] Key type lgssc registered [ 5254.055870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5258.519179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5261.865335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5264.968990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5268.275593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.669060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.705492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5271.714495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.834355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5272.854047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5272.902446] Lustre: lustre-MDT0000: new disk, initializing [ 5272.940792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.952299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5274.858876] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5280.512324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5280.549980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5280.595755] Lustre: 344025:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5280.630927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5280.634303] Lustre: Skipped 1 previous similar message [ 5280.676169] Lustre: lustre-MDT0001: new disk, initializing [ 5280.720487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5280.738795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5280.744416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5282.489480] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5286.471683] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5288.326752] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5291.488934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5291.531208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5291.682431] Lustre: lustre-OST0000: new disk, initializing [ 5291.686287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5291.732715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.259991] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5298.197584] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5298.697817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5298.704539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5298.757523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5300.001406] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5300.111419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5301.813103] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5301.922489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5303.780259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5303.780261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.782323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.786047] LustreError: Skipped 1 previous similar message [ 5303.793341] Lustre: Skipped 1 previous similar message [ 5304.882179] LustreError: 346966:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5304.941905] Lustre: server umount lustre-OST0000 complete [ 5308.898247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.899463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.906342] Lustre: Skipped 2 previous similar messages [ 5308.912814] Lustre: Skipped 1 previous similar message [ 5314.017308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.021259] Lustre: Skipped 1 previous similar message [ 5319.136892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.140850] Lustre: Skipped 1 previous similar message [ 5321.183264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5321.240781] LustreError: 347166:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5321.244613] LustreError: 347166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5321.284451] Lustre: server umount lustre-MDT0000 complete [ 5323.031820] LustreError: 345773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389700 with bad export cookie 11646692173543382099 [ 5323.036792] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5323.039281] LustreError: 345773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5323.142883] LustreError: 347371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5323.146768] LustreError: 347371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5323.207202] Lustre: server umount lustre-MDT0001 complete [ 5327.220825] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 16:35:02 (1767389702) [ 5330.084910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5332.787682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5343.021953] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5347.229631] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5351.501455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5354.464529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5357.306901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5359.313382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5361.317323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5361.338932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.412719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5361.423641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5361.458513] Lustre: lustre-MDT0000: new disk, initializing [ 5361.481177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.486742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5362.570234] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5366.300908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5366.322963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5366.351289] Lustre: 351139:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5366.370549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5366.372134] Lustre: Skipped 1 previous similar message [ 5366.405685] Lustre: lustre-MDT0001: new disk, initializing [ 5366.433693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5366.443319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5366.446167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5367.543655] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5370.333644] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5371.355473] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5373.301328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5373.322397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5375.019469] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5375.733974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5375.736859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5375.764920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5377.824520] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5378.985037] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5379.044782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5380.150966] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5380.207586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5381.090232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5381.090476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.092454] LustreError: Skipped 1 previous similar message [ 5381.095978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.098170] Lustre: Skipped 1 previous similar message [ 5386.862175] LustreError: 354075:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5386.863895] LustreError: 354075:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5386.895523] Lustre: server umount lustre-OST0000 complete [ 5391.328539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.329182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.333190] Lustre: Skipped 2 previous similar messages [ 5391.335208] Lustre: Skipped 2 previous similar messages [ 5394.116948] LustreError: 354277:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5394.119863] LustreError: 354277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5394.151223] Lustre: server umount lustre-MDT0000 complete [ 5395.638092] LustreError: 351131:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389773 with bad export cookie 11646692173543382960 [ 5395.643796] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.645542] LustreError: 351131:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5395.803666] Lustre: server umount lustre-MDT0001 complete [ 5399.651919] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:36:15 (1767389775) [ 5402.945540] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:36:18 (1767389778) [ 5406.869109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.122639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.126863] Lustre: Skipped 1 previous similar message [ 5408.793219] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5412.320884] LustreError: 355492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.331947] LustreError: 355492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5412.725954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5414.607922] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5417.152754] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5418.816744] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5422.095040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5424.685009] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5427.361423] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5436.918992] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5437.808203] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5438.681813] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5439.555341] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5440.480471] Lustre: Failing over lustre-MDT0000 [ 5440.723658] LustreError: 357932:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5440.725845] LustreError: 357932:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5440.753288] Lustre: server umount lustre-MDT0000 complete [ 5443.044099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.045034] LustreError: 356617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.052092] Lustre: Skipped 1 previous similar message [ 5443.064815] LustreError: 356617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5448.160981] LustreError: 356617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.171043] LustreError: 356617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5453.281598] LustreError: 356617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.294171] LustreError: 356617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5454.986961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.052195] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.173519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.178392] Lustre: Skipped 2 previous similar messages [ 5455.218297] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5456.873294] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5458.679715] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5460.466407] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5460.476655] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5460.507323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5462.223077] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5463.077185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5465.369238] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5466.336134] Lustre: Failing over lustre-MDT0000 [ 5466.584547] LustreError: 359016:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5466.586582] LustreError: 359016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5466.609117] Lustre: server umount lustre-MDT0000 complete [ 5468.939199] LustreError: 356618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.949946] LustreError: 356618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5470.690659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5470.691160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5470.703635] Lustre: Skipped 2 previous similar messages [ 5479.161989] LustreError: 357651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.169651] LustreError: 357651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5480.758188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.829502] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.014105] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5482.759244] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5483.792281] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5486.065955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5486.069851] Lustre: Skipped 2 previous similar messages [ 5486.077801] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5486.108258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5487.684976] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5488.451251] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5491.170762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.171405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5491.171769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.171776] Lustre: Skipped 1 previous similar message [ 5491.177030] Lustre: Skipped 2 previous similar messages [ 5497.326358] Lustre: server umount lustre-MDT0000 complete [ 5499.093875] LustreError: 357111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389876 with bad export cookie 11646692173543385620 [ 5499.100547] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5499.101337] LustreError: 357111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5499.196540] LustreError: 360252:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5499.200445] LustreError: 360252:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5499.254785] Lustre: server umount lustre-MDT0001 complete [ 5511.411641] Lustre: server umount lustre-OST0000 complete [ 5518.772674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.987331] LustreError: 361280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.028296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.032609] Lustre: Skipped 1 previous similar message [ 5520.823580] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5524.790667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5526.733839] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5529.313682] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5531.010215] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5534.235656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5536.977253] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5539.618583] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5543.660875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5549.027361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5549.029560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.031672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5549.031681] Lustre: Skipped 5 previous similar messages [ 5549.046609] Lustre: Skipped 2 previous similar messages [ 5552.806139] Lustre: server umount lustre-OST0000 complete [ 5560.749982] Lustre: server umount lustre-MDT0000 complete [ 5562.553148] LustreError: 361259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389940 with bad export cookie 11646692173543386362 [ 5562.559075] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5562.560633] LustreError: 361259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5562.706263] Lustre: server umount lustre-MDT0001 complete [ 5567.738294] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5569.189161] Key type lgssc unregistered [ 5569.342460] LNet: 364461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5569.347310] LNetError: 364461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5569.358499] LNet: Removed LNI 192.168.203.105@tcp [ 5569.795064] Key type .llcrypt unregistered [ 5569.797453] Key type ._llcrypt unregistered [ 5583.162349] Key type ._llcrypt registered [ 5583.164399] Key type .llcrypt registered [ 5583.227510] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5583.560661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5583.613587] alg: No test for adler32 (adler32-zlib) [ 5584.463603] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5584.562088] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5586.151154] Key type lgssc registered [ 5586.698065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5590.404120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5590.410775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.598247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.253433] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5597.253640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5597.444823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.124475] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5601.769220] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5603.555281] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5606.854293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5607.036310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5609.450686] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5612.001321] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5617.323743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5627.877414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.879371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5627.886210] Lustre: Skipped 2 previous similar messages [ 5627.890169] Lustre: Skipped 1 previous similar message [ 5632.849458] LustreError: 368207:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5632.886369] Lustre: server umount lustre-MDT0000 complete [ 5632.993134] LustreError: 365939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.000109] LustreError: 365939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5634.598896] LustreError: 365919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390012 with bad export cookie 17649405736274901042 [ 5634.603755] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5634.610043] LustreError: 365919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5634.714256] LustreError: 368409:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5634.718167] LustreError: 368409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5634.766499] Lustre: server umount lustre-MDT0001 complete [ 5646.844720] LustreError: 368610:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5646.848942] LustreError: 368610:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5646.914327] Lustre: server umount lustre-OST0000 complete [ 5653.286037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.528000] LustreError: 369340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.566645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.113080] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5658.592733] LustreError: 369340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.957396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5659.127410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5660.846968] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5663.491523] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5665.257486] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5668.393247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5668.553175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.953988] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5673.508438] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5678.828827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5689.315814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.317867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5689.323820] Lustre: Skipped 2 previous similar messages [ 5689.327909] Lustre: Skipped 2 previous similar messages [ 5692.171801] LustreError: 371688:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5692.175791] LustreError: 371688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5692.211887] Lustre: server umount lustre-MDT0000 complete [ 5693.911898] LustreError: 369321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390071 with bad export cookie 17649405736274902155 [ 5693.917895] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.919617] LustreError: 369321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5694.069308] Lustre: server umount lustre-MDT0001 complete [ 5706.103783] LustreError: 372091:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5706.107594] LustreError: 372091:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5706.168967] Lustre: server umount lustre-OST0000 complete [ 5713.218107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.418455] LustreError: 372919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.445358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.161091] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5718.496832] LustreError: 372919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.040041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5720.725188] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5723.051579] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5724.675971] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5727.925118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5730.526553] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5733.171102] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5738.348183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5743.589271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.589846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.591427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5743.597242] Lustre: Skipped 1 previous similar message [ 5746.162171] LustreError: 375167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5746.166256] LustreError: 375167:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5746.223291] Lustre: server umount lustre-OST0000 complete [ 5748.706144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5748.715417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.719326] Lustre: Skipped 2 previous similar messages [ 5753.825224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.829248] Lustre: Skipped 1 previous similar message [ 5754.157727] Lustre: server umount lustre-MDT0000 complete [ 5755.896176] LustreError: 372900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390133 with bad export cookie 17649405736274903163 [ 5755.901766] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.903556] LustreError: 372900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5756.051538] Lustre: server umount lustre-MDT0001 complete [ 5760.766310] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5762.185386] Key type lgssc unregistered [ 5762.355703] LNet: 376103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5762.360732] LNetError: 376103:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5762.369655] LNet: Removed LNI 192.168.203.105@tcp [ 5762.802397] Key type .llcrypt unregistered [ 5762.804738] Key type ._llcrypt unregistered [ 5773.836669] Key type ._llcrypt registered [ 5773.838292] Key type .llcrypt registered [ 5773.886296] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5774.327209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5774.335532] alg: No test for adler32 (adler32-zlib) [ 5775.229967] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5775.353210] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5776.951249] Key type lgssc registered [ 5777.571753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5781.060173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5781.070158] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.075820] LustreError: 377500:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5781.087567] LustreError: 377500:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5781.092044] LustreError: 377500:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5781.096592] LustreError: 377500:0:(tgt_mount.c:2540:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5781.101750] LustreError: 377500:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5784.753147] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5784.758312] LustreError: 377897:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5784.772750] LustreError: 377897:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5784.777460] LustreError: 377897:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5784.783092] LustreError: 377897:0:(tgt_mount.c:2540:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5784.789744] LustreError: 377897:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5793.507172] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5795.085472] Key type lgssc unregistered [ 5795.257794] LNet: 378883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5795.262798] LNetError: 378883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5795.277704] LNet: Removed LNI 192.168.203.105@tcp [ 5795.742928] Key type .llcrypt unregistered [ 5795.745264] Key type ._llcrypt unregistered [ 5804.289320] Key type ._llcrypt registered [ 5804.291625] Key type .llcrypt registered [ 5804.348867] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5812.439065] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5812.753826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5812.796512] alg: No test for adler32 (adler32-zlib) [ 5813.655492] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5813.738147] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5815.335203] Key type lgssc registered [ 5815.877934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5820.112193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5823.352461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5826.235536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.436091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5832.723941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5832.751207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5832.760651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.867125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5833.883386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5833.937728] Lustre: lustre-MDT0000: new disk, initializing [ 5833.971575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.983031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5835.580218] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5841.106086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5841.145233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5841.199567] Lustre: 381990:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5841.239501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5841.242801] Lustre: Skipped 1 previous similar message [ 5841.284483] Lustre: lustre-MDT0001: new disk, initializing [ 5841.316111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5841.330789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5841.334831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5842.960456] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5846.900319] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5848.543893] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5851.763361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5851.805459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5851.947499] Lustre: lustre-OST0000: new disk, initializing [ 5851.951475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5851.996718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5854.509579] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5857.794649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5857.800129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5857.855183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5858.524265] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5860.383293] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5860.481335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5862.256907] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5862.346132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5868.005304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.006420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.010233] LustreError: Skipped 1 previous similar message [ 5868.019282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.025870] Lustre: Skipped 1 previous similar message [ 5869.360383] LustreError: 384927:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5869.423665] Lustre: server umount lustre-OST0000 complete [ 5872.096401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5872.101719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.109611] Lustre: Skipped 1 previous similar message [ 5872.113379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.121562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.122438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.392993] LustreError: 385129:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5877.397596] LustreError: 385129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5877.437925] Lustre: server umount lustre-MDT0000 complete [ 5879.221095] LustreError: 381980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390256 with bad export cookie 667964515574723530 [ 5879.228166] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5879.232035] LustreError: 381980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5879.321790] LustreError: 385333:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5879.326047] LustreError: 385333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5879.382467] Lustre: server umount lustre-MDT0001 complete [ 5883.486507] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5884.370824] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:44:19 (1767390259) [ 5887.438687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.647862] LustreError: 386053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.657772] LustreError: 386053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5887.683772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.382628] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5893.088901] LustreError: 386053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.339423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5893.544455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5895.310852] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5897.973871] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5899.814334] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5903.092629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5903.243895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.778709] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5908.408716] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5911.746705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.788341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5911.871933] Lustre: lustre-OST0001: new disk, initializing [ 5911.875638] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5914.443529] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5918.512473] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5918.740801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5918.747894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5918.798773] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5923.812893] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5923.814761] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.817658] LustreError: Skipped 1 previous similar message [ 5923.826251] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5923.832024] Lustre: Skipped 2 previous similar messages [ 5926.834105] LustreError: 389256:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5926.838267] LustreError: 389256:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5926.867573] Lustre: server umount lustre-OST0001 complete [ 5928.933936] LustreError: 387649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.943667] LustreError: 387649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5934.054308] LustreError: 389167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.064365] LustreError: 389167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5939.180764] LustreError: 387650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.191352] LustreError: 387650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5943.777577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5943.778276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.782532] LustreError: Skipped 1 previous similar message [ 5943.790430] Lustre: Skipped 1 previous similar message [ 5943.791296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5943.799869] Lustre: Skipped 2 previous similar messages [ 5945.648222] LustreError: 389479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5945.651364] LustreError: 389479:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5945.704561] Lustre: server umount lustre-OST0000 complete [ 5953.680162] LustreError: 389680:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5953.684441] LustreError: 389680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5953.714785] Lustre: server umount lustre-MDT0000 complete [ 5954.528799] LustreError: 386053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.539765] LustreError: 386053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5955.452435] LustreError: 386034:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390332 with bad export cookie 667964515574724398 [ 5955.459509] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.459810] LustreError: 386034:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5961.680131] Lustre: server umount lustre-MDT0001 complete [ 5966.484430] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5967.942269] Key type lgssc unregistered [ 5968.111739] LNet: 390415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5968.116466] LNetError: 390415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5968.126618] LNet: Removed LNI 192.168.203.105@tcp [ 5968.573261] Key type .llcrypt unregistered [ 5968.575522] Key type ._llcrypt unregistered [ 5983.742697] Key type ._llcrypt registered [ 5983.744566] Key type .llcrypt registered [ 5983.799456] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5992.011627] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5992.322871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5992.359277] alg: No test for adler32 (adler32-zlib) [ 5993.217074] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5993.293642] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5994.871151] Key type lgssc registered [ 5995.415117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5999.584198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.824451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6005.731715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6008.965843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.363473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.399331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6012.407939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.524287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6013.544944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6013.604237] Lustre: lustre-MDT0000: new disk, initializing [ 6013.644849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.657293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6015.458832] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6020.947522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.992137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6021.048564] Lustre: 394336:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6021.114073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6021.117000] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6021.119833] Lustre: Skipped 1 previous similar message [ 6021.159389] Lustre: lustre-MDT0001: new disk, initializing [ 6021.201902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6021.217958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6021.222116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6022.809848] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6026.789726] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6028.626693] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6032.026621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.064820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6032.210203] Lustre: lustre-OST0000: new disk, initializing [ 6032.212856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6032.265505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.940240] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6036.303372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6036.310149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6036.370084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6038.965803] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6040.946764] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6041.072775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6042.920993] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6043.053205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6046.696689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.700046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.701593] LustreError: Skipped 1 previous similar message [ 6046.711038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.715993] Lustre: Skipped 1 previous similar message [ 6050.098233] LustreError: 397279:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6050.158318] Lustre: server umount lustre-OST0000 complete [ 6056.932318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.932517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6056.933356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6056.940366] Lustre: Skipped 2 previous similar messages [ 6058.127656] LustreError: 397481:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6058.131900] LustreError: 397481:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6058.169008] Lustre: server umount lustre-MDT0000 complete [ 6059.880043] LustreError: 395118:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390437 with bad export cookie 2546007427501029592 [ 6059.882789] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6059.883051] LustreError: 395118:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6059.946125] LustreError: 397686:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6059.947833] LustreError: 397686:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6059.991307] Lustre: server umount lustre-MDT0001 complete [ 6063.705392] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 16:47:19 (1767390439) [ 6067.676555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6071.580647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.771422] Lustre: lustre-MDT0000: new disk, initializing [ 6071.820541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.673345] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6076.896980] LustreError: 398951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.902747] LustreError: 398951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6077.670404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6077.842931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.627782] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6081.947190] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6083.676892] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6087.138412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.139357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.146452] Lustre: Skipped 1 previous similar message [ 6087.153080] Lustre: Skipped 2 previous similar messages [ 6090.833135] LustreError: 400292:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6090.839318] LustreError: 400292:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6090.875795] Lustre: server umount lustre-MDT0000 complete [ 6092.589062] LustreError: 398932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390470 with bad export cookie 2546007427501030474 [ 6092.595053] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.596593] LustreError: 398932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6092.740650] Lustre: server umount lustre-MDT0001 complete [ 6097.256872] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:47:52 (1767390472) [ 6100.368307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.590649] LustreError: 401216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.627274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.397959] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6106.081461] LustreError: 401216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.440714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6108.350951] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6110.991938] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6112.815467] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6116.035922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6118.891462] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6121.479962] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6130.130608] Lustre: Permanently deactivating lustre-OST0000 [ 6130.140619] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6139.898490] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6140.990480] LustreError: 404005:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6140.997685] LustreError: 404005:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6141.002594] LustreError: 404005:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6141.847215] LustreError: 404053:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6141.852269] LustreError: 404053:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6148.850125] LustreError: 404150:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6148.854228] LustreError: 404150:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6148.909759] Lustre: server umount lustre-OST0000 complete [ 6149.901213] LustreError: 404303:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6149.906214] LustreError: 404303:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6152.672547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6152.677609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.686360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6152.690138] Lustre: Skipped 2 previous similar messages [ 6161.379778] LustreError: 404400:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6161.383154] LustreError: 404400:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6161.417908] Lustre: server umount lustre-MDT0000 complete [ 6163.116985] LustreError: 402835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390540 with bad export cookie 2546007427501031013 [ 6163.122444] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6163.123884] LustreError: 402835:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6172.713789] Lustre: server umount lustre-MDT0001 complete [ 6176.467718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.162817] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6179.420478] LustreError: 405444:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6182.261474] Lustre: 405783:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.105@tcp [ 6182.871464] LustreError: 405880:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6182.873558] LustreError: 405880:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6182.903478] Lustre: server umount MGS complete [ 6187.873312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.068459] LustreError: 406508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.074905] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6188.079772] Lustre: Skipped 1 previous similar message [ 6188.111897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.116385] Lustre: Skipped 2 previous similar messages [ 6189.866095] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6193.121068] LustreError: 406508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.840087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6193.969291] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6195.738262] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6198.346037] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6200.141438] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6201.026210] Lustre: Permanently reactivating lustre-OST0000 [ 6201.035267] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6201.039678] Lustre: Skipped 2 previous similar messages [ 6226.911169] LustreError: 408063:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6230.077944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6230.233267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.235962] Lustre: Skipped 1 previous similar message [ 6232.732692] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6235.266501] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6247.908594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.908912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.910341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.910357] Lustre: Skipped 4 previous similar messages [ 6247.917171] Lustre: Skipped 2 previous similar messages [ 6262.239182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6262.261271] LustreError: 409799:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6262.264409] LustreError: 409799:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6262.311309] Lustre: server umount lustre-OST0000 complete [ 6265.824350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6265.829235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.837796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.841774] Lustre: Skipped 5 previous similar messages [ 6270.188109] Lustre: server umount lustre-MDT0000 complete [ 6271.940081] LustreError: 406489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390649 with bad export cookie 2546007427501031972 [ 6271.945766] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.947309] LustreError: 406489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6272.104555] Lustre: server umount lustre-MDT0001 complete [ 6276.936479] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6278.392512] Key type lgssc unregistered [ 6278.557665] LNet: 410738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6278.562392] LNetError: 410738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6278.570593] LNet: Removed LNI 192.168.203.105@tcp [ 6278.946395] Key type .llcrypt unregistered [ 6278.948537] Key type ._llcrypt unregistered [ 6287.078993] Key type ._llcrypt registered [ 6287.080491] Key type .llcrypt registered [ 6287.128523] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 6295.076984] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6295.424963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6295.472795] alg: No test for adler32 (adler32-zlib) [ 6296.359225] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6296.467289] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6298.055168] Key type lgssc registered [ 6298.614966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.948695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6306.148705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6308.975737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.775577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6314.958948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6314.991118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6314.999703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.111028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6316.124438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6316.165763] Lustre: lustre-MDT0000: new disk, initializing [ 6316.190153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6316.197632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6317.953471] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6323.499624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6323.534681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6323.588942] Lustre: 413843:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6323.630928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6323.634867] Lustre: Skipped 1 previous similar message [ 6323.687721] Lustre: lustre-MDT0001: new disk, initializing [ 6323.728334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6323.745702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6323.751151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6325.454749] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6329.087408] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6330.928592] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6334.198027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6334.238428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6334.381252] Lustre: lustre-OST0000: new disk, initializing [ 6334.384231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6334.423748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.803898] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6340.908354] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6342.655526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6342.659596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6342.700096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6342.769153] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6342.856074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6344.522585] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6344.640240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6347.747584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.756742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6347.756909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6347.764991] Lustre: Skipped 1 previous similar message [ 6351.601137] LustreError: 416780:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6351.662308] Lustre: server umount lustre-OST0000 complete [ 6354.400569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6354.405620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.413241] Lustre: Skipped 1 previous similar message [ 6354.416785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.985105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.985909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.632060] LustreError: 416982:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6359.636183] LustreError: 416982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6359.669074] Lustre: server umount lustre-MDT0000 complete [ 6361.424756] LustreError: 414622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390738 with bad export cookie 17612504615739695842 [ 6361.430445] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6361.432350] LustreError: 414622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6361.523788] LustreError: 417184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6361.526473] LustreError: 417184:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6361.574873] Lustre: server umount lustre-MDT0001 complete [ 6365.716638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6366.590948] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:52:22 (1767390742) [ 6369.839334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.103979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.889915] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6375.393126] LustreError: 417902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.403038] LustreError: 417902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6375.899332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6376.073386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6377.660704] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6380.088923] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6381.601230] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6384.802120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6384.950873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.446579] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6390.076668] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6391.021159] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6396.792692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6400.483787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6400.485587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6400.490426] Lustre: Skipped 1 previous similar message [ 6400.490858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6400.493676] Lustre: Skipped 2 previous similar messages [ 6404.657853] LustreError: 420339:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6404.661993] LustreError: 420339:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6404.717909] Lustre: server umount lustre-OST0000 complete [ 6407.136424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6407.141324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.149779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.620653] LustreError: 420541:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6412.624738] LustreError: 420541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6412.657824] Lustre: server umount lustre-MDT0000 complete [ 6414.392932] LustreError: 417882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390791 with bad export cookie 17612504615739696710 [ 6414.398554] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.400301] LustreError: 417882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6414.541312] Lustre: server umount lustre-MDT0001 complete [ 6419.389248] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6420.844369] Key type lgssc unregistered [ 6421.013787] LNet: 421275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6421.018494] LNetError: 421275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6421.028569] LNet: Removed LNI 192.168.203.105@tcp [ 6421.470229] Key type .llcrypt unregistered [ 6421.472518] Key type ._llcrypt unregistered [ 6431.988526] Key type ._llcrypt registered [ 6431.990086] Key type .llcrypt registered [ 6432.038625] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6432.190316] Key type .llcrypt unregistered [ 6432.192581] Key type ._llcrypt unregistered [ 6441.342380] Key type ._llcrypt registered [ 6441.344632] Key type .llcrypt registered [ 6441.411328] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6441.842684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6441.878556] alg: No test for adler32 (adler32-zlib) [ 6442.787054] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6442.909962] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6444.519161] Key type lgssc registered [ 6445.131069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6449.087770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6449.099122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6450.360742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6452.161149] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6456.168825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6456.331915] LustreError: 424152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6456.336576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.340498] LustreError: 424152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6458.793739] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6460.450659] LustreError: 423425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.458326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6460.461505] LustreError: 423425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6461.302821] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6464.596559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6464.783298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.501732] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6475.237482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.240942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.245683] Lustre: Skipped 1 previous similar message [ 6475.252976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6480.355403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6480.359542] Lustre: Skipped 1 previous similar message [ 6480.690162] LustreError: 425478:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6480.745732] Lustre: server umount lustre-OST0000 complete [ 6484.963352] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6484.968135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.976159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6484.979733] Lustre: Skipped 1 previous similar message [ 6488.653538] LustreError: 425681:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6488.657708] LustreError: 425681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6488.692655] Lustre: server umount lustre-MDT0000 complete [ 6490.440841] LustreError: 423405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390867 with bad export cookie 2475651681309250104 [ 6490.447814] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6490.448462] LustreError: 423405:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6490.542453] LustreError: 425884:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6490.545043] LustreError: 425884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6490.591493] Lustre: server umount lustre-MDT0001 complete [ 6495.493896] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6496.949213] Key type lgssc unregistered [ 6497.118595] LNet: 426415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6497.123552] LNetError: 426415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6497.132555] LNet: Removed LNI 192.168.203.105@tcp [ 6497.592324] Key type .llcrypt unregistered [ 6497.594615] Key type ._llcrypt unregistered [ 6509.851857] Key type ._llcrypt registered [ 6509.854292] Key type .llcrypt registered [ 6509.913510] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6510.267647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6510.298703] alg: No test for adler32 (adler32-zlib) [ 6511.161630] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6511.243754] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6512.831164] Key type lgssc registered [ 6513.391284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6517.182325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6517.193685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6532.639340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6537.759180] LustreError: 427761:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 6564.383197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.947748] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6569.587609] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6572.821033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.111439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.157049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6574.818392] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6578.146938] LustreError: 427788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.155295] LustreError: 427788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6578.903645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6579.103400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6580.926619] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6583.572644] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6585.390379] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6588.384171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6588.389033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.397304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6589.408982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6589.413941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.422523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6590.435727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6590.439679] Lustre: Skipped 1 previous similar message [ 6594.354193] LustreError: 430049:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6594.409797] Lustre: server umount lustre-OST0000 complete [ 6599.139286] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6599.144078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.152488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.316374] LustreError: 430253:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6602.320501] LustreError: 430253:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6602.352369] Lustre: server umount lustre-MDT0000 complete [ 6604.099737] LustreError: 427770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767390981 with bad export cookie 14641730324944276714 [ 6604.106154] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.107129] LustreError: 427770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6604.190326] LustreError: 430455:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6604.193340] LustreError: 430455:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6604.260829] Lustre: server umount lustre-MDT0001 complete [ 6609.162248] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6610.563582] Key type lgssc unregistered [ 6610.720673] LNet: 430986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6610.725533] LNetError: 430986:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6610.742496] LNet: Removed LNI 192.168.203.105@tcp [ 6611.188269] Key type .llcrypt unregistered [ 6611.190590] Key type ._llcrypt unregistered [ 6623.039920] Key type ._llcrypt registered [ 6623.041788] Key type .llcrypt registered [ 6623.102249] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6623.495629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6623.515830] alg: No test for adler32 (adler32-zlib) [ 6624.369718] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6624.443784] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6626.023157] Key type lgssc registered [ 6626.593648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6630.325542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6630.337355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.515876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.291556] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6637.309214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6637.521429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.395667] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6642.029812] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6643.883814] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6647.234758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6647.415397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6650.195892] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6652.864395] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6655.600914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6661.091825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.094950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6661.099592] Lustre: Skipped 2 previous similar messages [ 6661.103484] Lustre: Skipped 1 previous similar message [ 6662.114236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.118078] Lustre: Skipped 1 previous similar message [ 6664.453562] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 6666.512420] LustreError: 434607:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6666.556954] Lustre: server umount lustre-MDT0000 complete [ 6667.236807] LustreError: 433426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6667.247063] LustreError: 433426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6667.691814] LustreError: 432342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.5@tcp arrived at 1767391045 with bad export cookie 11671443054410374710 [ 6667.700488] LustreError: 432342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6668.532911] LustreError: 432342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767391046 with bad export cookie 11671443054410374080 [ 6668.538562] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6672.353565] LustreError: 432361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.355327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.363529] LustreError: 432361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6672.372107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6673.376698] LustreError: 432361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.386547] LustreError: 432361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6674.610190] LustreError: 434812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6674.614355] LustreError: 434812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6674.676926] Lustre: server umount lustre-OST0000 complete [ 6678.495441] LustreError: 433073:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001e09c507 x1853243786291584/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6678.514827] LustreError: 435166:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6678.518943] LustreError: 435166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6678.606271] Lustre: server umount lustre-MDT0001 complete [ 6683.785745] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6685.395295] Key type lgssc unregistered [ 6685.583799] LNet: 435697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6685.588678] LNetError: 435697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6685.598757] LNet: Removed LNI 192.168.203.105@tcp [ 6686.092501] Key type .llcrypt unregistered [ 6686.094778] Key type ._llcrypt unregistered [ 6698.163123] Key type ._llcrypt registered [ 6698.164655] Key type .llcrypt registered [ 6698.228615] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6698.647977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6698.703023] alg: No test for adler32 (adler32-zlib) [ 6699.613506] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6699.729230] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6701.343207] Key type lgssc registered [ 6701.859672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6705.747547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6705.760157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.989229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6708.920225] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6713.150135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6713.344090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6715.212967] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6717.908944] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6719.755269] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6723.118847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6723.327357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.080814] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6728.710721] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6731.564491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6736.868165] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.877791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6736.881682] Lustre: Skipped 1 previous similar message [ 6740.217305] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [ 6741.988389] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6741.992536] Lustre: Skipped 1 previous similar message [ 6742.223244] LustreError: 439319:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6742.250033] Lustre: server umount lustre-MDT0001 complete [ 6747.106101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6747.106267] LustreError: 437785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6747.107516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6747.107531] Lustre: Skipped 1 previous similar message [ 6747.108965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6747.135464] LustreError: 437785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6750.642148] LustreError: 439523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6750.646316] LustreError: 439523:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6750.703080] Lustre: server umount lustre-OST0000 complete [ 6752.226421] LustreError: 437785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.574212] LustreError: 439724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6752.578235] LustreError: 439724:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6752.646724] Lustre: server umount lustre-MDT0000 complete [ 6758.895661] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6760.390287] Key type lgssc unregistered [ 6760.560752] LNet: 440407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6760.565780] LNetError: 440407:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6760.576586] LNet: Removed LNI 192.168.203.105@tcp [ 6761.001708] Key type .llcrypt unregistered [ 6761.004048] Key type ._llcrypt unregistered [ 6772.362881] Key type ._llcrypt registered [ 6772.364893] Key type .llcrypt registered [ 6772.414205] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 6780.472784] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6780.898185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6780.909178] alg: No test for adler32 (adler32-zlib) [ 6781.794957] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6781.908488] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6783.511165] Key type lgssc registered [ 6784.088524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6788.462905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.357119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6794.250380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6797.391514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.979746] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6806.196783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6809.478222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6809.513428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6809.522791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.634847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6810.652634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6810.709310] Lustre: lustre-MDT0000: new disk, initializing [ 6810.750239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.762788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6812.488197] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6817.254714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6820.489300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6820.533957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6820.679426] Lustre: lustre-OST0000: new disk, initializing [ 6820.683137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6820.686703] Lustre: Skipped 1 previous similar message [ 6820.721633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.310561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6823.315450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6823.322397] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6823.329206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6827.383528] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6831.724669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6834.991612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6835.030307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6835.092628] Lustre: 444356:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6835.140710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6835.191696] Lustre: lustre-MDT0001: new disk, initializing [ 6835.245243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6835.264284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6835.268748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6836.893251] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6841.381280] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6841.501225] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6843.240699] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6843.355990] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6846.212812] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [ 6846.947723] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.948883] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6846.951236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6846.957291] Lustre: Skipped 1 previous similar message [ 6851.321282] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [ 6851.326880] Lustre: Skipped 1 previous similar message [ 6851.727338] LustreError: 447776:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6851.761981] Lustre: server umount lustre-MDT0001 complete [ 6857.187382] LustreError: 445371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.197963] LustreError: 445371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6862.307722] LustreError: 444361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.319218] LustreError: 444361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6867.426178] LustreError: 444367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.426552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.445308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6872.545256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6872.546598] LustreError: 444367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.666842] LustreError: 444366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6879.199190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6879.229162] LustreError: 448050:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6879.233704] LustreError: 448050:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6879.299146] Lustre: server umount lustre-OST0000 complete [ 6887.312400] LustreError: 448253:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6887.316904] LustreError: 448253:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6887.396808] Lustre: server umount lustre-MDT0000 complete [ 6893.353689] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6894.828392] Key type lgssc unregistered [ 6894.958661] LNet: 448949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6894.963603] LNetError: 448949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6894.974568] LNet: Removed LNI 192.168.203.105@tcp [ 6895.421744] Key type .llcrypt unregistered [ 6895.423992] Key type ._llcrypt unregistered [ 6913.207929] Key type ._llcrypt registered [ 6913.209586] Key type .llcrypt registered [ 6913.267293] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 6913.429297] Key type .llcrypt unregistered [ 6913.431452] Key type ._llcrypt unregistered [ 6921.596564] Key type ._llcrypt registered [ 6921.598468] Key type .llcrypt registered [ 6921.658350] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 6922.111605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.121457] alg: No test for adler32 (adler32-zlib) [ 6922.985125] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 6923.089064] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 6924.679154] Key type lgssc registered [ 6925.270251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6929.209706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6932.357386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6935.367047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.580122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.613462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6938.623066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.731822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6939.748810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6939.793345] Lustre: lustre-MDT0000: new disk, initializing [ 6939.827453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.839104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6941.656116] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6947.215358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.250725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6947.305775] Lustre: 452268:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6947.345299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6947.349225] Lustre: Skipped 1 previous similar message [ 6947.395507] Lustre: lustre-MDT0001: new disk, initializing [ 6947.443070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6947.466189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6947.472964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6949.182691] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6953.086621] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6954.820736] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6958.138351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6958.179913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6958.337245] Lustre: lustre-OST0000: new disk, initializing [ 6958.340770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6958.385697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.964520] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 6964.233287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6964.239664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6964.293677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6964.987123] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6969.316208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.318257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6969.324107] Lustre: Skipped 2 previous similar messages [ 6969.328084] Lustre: Skipped 1 previous similar message [ 6973.967914] LustreError: 454712:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6974.004406] Lustre: server umount lustre-MDT0000 complete [ 6975.740687] LustreError: 453048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767391353 with bad export cookie 16192711410453866157 [ 6975.746191] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.748235] LustreError: 453048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6975.842989] LustreError: 454915:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6975.846557] LustreError: 454915:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6975.908221] Lustre: server umount lustre-MDT0001 complete [ 6987.894562] LustreError: 455116:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6987.898734] LustreError: 455116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6987.968287] Lustre: server umount lustre-OST0000 complete [ 6992.942196] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:02:48 (1767391368) [ 6993.451946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6996.733273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.958559] LustreError: 455942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.970769] LustreError: 455942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6996.997583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6998.634624] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7002.081849] LustreError: 455942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.704675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7002.860130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.597001] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7007.266751] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7009.056493] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7012.349041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7012.387916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7012.490248] Lustre: Found index 0 for lustre-OST0000, updating log [ 7012.587227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.205630] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7017.825150] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7021.803247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7027.172514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.174208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7027.175677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7027.175686] Lustre: Skipped 1 previous similar message [ 7027.180515] Lustre: Skipped 1 previous similar message [ 7028.258701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7028.262609] Lustre: Skipped 2 previous similar messages [ 7030.899439] LustreError: 458205:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7030.903601] LustreError: 458205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7030.960215] Lustre: server umount lustre-OST0000 complete [ 7032.292968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7033.313139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.314154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.320456] Lustre: Skipped 1 previous similar message [ 7039.373075] LustreError: 458459:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7039.376984] LustreError: 458459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7039.413302] Lustre: server umount lustre-MDT0000 complete [ 7039.456669] LustreError: 455941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.466610] LustreError: 455941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7041.214989] LustreError: 455921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767391418 with bad export cookie 16192711410453867550 [ 7041.219694] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7041.222363] LustreError: 455921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7041.368842] Lustre: server umount lustre-MDT0001 complete [ 7045.312380] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:03:40 (1767391420) [ 7045.807266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7046.282602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7046.761132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7047.242475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7047.727372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.216607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7048.706738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.192302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.670217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7050.150459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7050.617281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7051.079360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7051.544761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7052.014089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7052.488630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7052.962462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.441439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7053.924063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7054.399347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7054.883398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7055.339339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7055.822297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7056.300713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7056.784319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.244199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7057.719446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7058.210139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7058.695300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7059.185667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7059.677739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7060.174682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7060.652094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7061.118321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7061.594363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7062.061716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7062.543947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7063.024131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7063.511549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7063.995738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.479450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7064.964845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7065.444624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7065.934290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7066.420636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7066.894554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7067.366361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7067.863817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.358518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.845428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7069.320396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7069.796558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7072.607740] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp [ 7073.473643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.686974] LustreError: 462047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.727461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.339960] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7078.880993] LustreError: 462047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.363377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7081.311448] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7083.848823] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7085.562311] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7088.819178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7088.865073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7088.984957] Lustre: Found index 0 for lustre-OST0000, updating log [ 7091.335366] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7093.616520] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7095.267107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7095.269992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.276701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7095.279858] Lustre: Skipped 3 previous similar messages [ 7096.625157] LustreError: 464358:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7096.629203] LustreError: 464358:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7096.683197] Lustre: server umount lustre-OST0000 complete [ 7099.490407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.498214] Lustre: Skipped 1 previous similar message [ 7103.585357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.589268] Lustre: Skipped 9 previous similar messages [ 7104.610291] LustreError: 462046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.619068] LustreError: 462046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7104.620830] Lustre: server umount lustre-MDT0000 complete [ 7106.372771] LustreError: 462027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767391483 with bad export cookie 16192711410453868621 [ 7106.379498] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7106.380171] LustreError: 462027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7106.539169] Lustre: server umount lustre-MDT0001 complete [ 7107.824786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7115.075453] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 7119.587152] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7123.956376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7127.190372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7130.257377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7133.444736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7135.072125] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp [ 7136.818248] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:05:12 (1767391512) [ 7153.243812] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7154.734258] Key type lgssc unregistered [ 7154.896890] LNet: 468344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7154.901886] LNetError: 468344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7283.943875] LNet: Removed LNI 192.168.203.105@tcp [ 7284.404938] Key type .llcrypt unregistered [ 7284.407199] Key type ._llcrypt unregistered [ 7293.142434] Key type ._llcrypt registered [ 7293.144389] Key type .llcrypt registered [ 7293.185280] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7293.449530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7293.455216] alg: No test for adler32 (adler32-zlib) [ 7294.313204] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 7294.409521] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7295.991155] Key type lgssc registered [ 7296.536716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7298.531214] LNet: Added LNI 192.168.203.105@tcp100 [8/256/0/180] [ 7298.952395] LNet: Added LNI 192.168.203.105@tcp101 [8/256/0/180] [ 7299.393689] LNet: Added LNI 192.168.203.105@tcp102 [8/256/0/180] [ 7299.825455] LNet: Added LNI 192.168.203.105@tcp103 [8/256/0/180] [ 7300.259116] LNet: Added LNI 192.168.203.105@tcp104 [8/256/0/180] [ 7300.685658] LNet: Added LNI 192.168.203.105@tcp105 [8/256/0/180] [ 7301.094592] LNet: Added LNI 192.168.203.105@tcp106 [8/256/0/180] [ 7301.533057] LNet: Added LNI 192.168.203.105@tcp107 [8/256/0/180] [ 7301.963833] LNet: Added LNI 192.168.203.105@tcp108 [8/256/0/180] [ 7302.389646] LNet: Added LNI 192.168.203.105@tcp109 [8/256/0/180] [ 7302.826692] LNet: Added LNI 192.168.203.105@tcp110 [8/256/0/180] [ 7303.229256] LNet: Added LNI 192.168.203.105@tcp111 [8/256/0/180] [ 7303.638631] LNet: Added LNI 192.168.203.105@tcp112 [8/256/0/180] [ 7304.066430] LNet: Added LNI 192.168.203.105@tcp113 [8/256/0/180] [ 7304.486631] LNet: Added LNI 192.168.203.105@tcp114 [8/256/0/180] [ 7304.909741] LNet: Added LNI 192.168.203.105@tcp115 [8/256/0/180] [ 7305.338451] LNet: Added LNI 192.168.203.105@tcp116 [8/256/0/180] [ 7305.765838] LNet: Added LNI 192.168.203.105@tcp117 [8/256/0/180] [ 7306.183536] LNet: Added LNI 192.168.203.105@tcp118 [8/256/0/180] [ 7306.611579] LNet: Added LNI 192.168.203.105@tcp119 [8/256/0/180] [ 7307.031121] LNet: Added LNI 192.168.203.105@tcp120 [8/256/0/180] [ 7307.455803] LNet: Added LNI 192.168.203.105@tcp121 [8/256/0/180] [ 7307.877302] LNet: Added LNI 192.168.203.105@tcp122 [8/256/0/180] [ 7308.301382] LNet: Added LNI 192.168.203.105@tcp123 [8/256/0/180] [ 7308.712618] LNet: Added LNI 192.168.203.105@tcp124 [8/256/0/180] [ 7309.123132] LNet: Added LNI 192.168.203.105@tcp125 [8/256/0/180] [ 7309.562297] LNet: Added LNI 192.168.203.105@tcp126 [8/256/0/180] [ 7309.989747] LNet: Added LNI 192.168.203.105@tcp127 [8/256/0/180] [ 7310.415312] LNet: Added LNI 192.168.203.105@tcp128 [8/256/0/180] [ 7310.828066] LNet: Added LNI 192.168.203.105@tcp129 [8/256/0/180] [ 7311.245530] LNet: Added LNI 192.168.203.105@tcp130 [8/256/0/180] [ 7311.671207] LNet: Added LNI 192.168.203.105@tcp131 [8/256/0/180] [ 7312.094658] LNet: Added LNI 192.168.203.105@tcp132 [8/256/0/180] [ 7312.504054] LNet: Added LNI 192.168.203.105@tcp133 [8/256/0/180] [ 7312.932257] LNet: Added LNI 192.168.203.105@tcp134 [8/256/0/180] [ 7313.356704] LNet: Added LNI 192.168.203.105@tcp135 [8/256/0/180] [ 7316.637642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7316.669299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7316.678890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.730651] Lustre: 471334:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [ 7317.794036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7317.809341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7317.854281] Lustre: lustre-MDT0000: new disk, initializing [ 7317.891716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.902910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7319.621609] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7325.152657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7325.190082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7325.260567] Lustre: 471352:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7325.304122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7325.307691] Lustre: Skipped 1 previous similar message [ 7325.352147] Lustre: lustre-MDT0001: new disk, initializing [ 7325.366500] Lustre: 472140:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [ 7325.370546] Lustre: 472140:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7325.433088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7325.455318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7325.462051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7327.247221] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7331.298667] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7333.098942] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7333.889196] Lustre: 472791:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [ 7333.895486] Lustre: 472791:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7336.380147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7336.416081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7336.582096] Lustre: lustre-OST0000: new disk, initializing [ 7336.586148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7336.645196] Lustre: 473111:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [ 7336.651222] Lustre: 473111:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7336.680951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.261954] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7341.567729] Lustre: 473590:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.105@tcp: can't add NID 192.168.203.105@tcp130: rc = -75 [ 7341.574191] Lustre: 473590:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7341.608662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7341.614409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7341.687730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7343.232705] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7346.659629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.659680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7346.668643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7346.676078] Lustre: Skipped 1 previous similar message [ 7348.530155] LustreError: 473946:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7348.589568] Lustre: server umount lustre-OST0000 complete [ 7351.264503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7351.269645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.277684] Lustre: Skipped 1 previous similar message [ 7351.281219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7356.495197] LustreError: 474147:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7356.499225] LustreError: 474147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7356.535861] Lustre: server umount lustre-MDT0000 complete [ 7358.294981] LustreError: 471345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767391735 with bad export cookie 3376186520485324971 [ 7358.302181] LustreError: 471345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7358.302490] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.405513] LustreError: 474349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7358.408805] LustreError: 474349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7358.469562] Lustre: server umount lustre-MDT0001 complete [ 7363.321719] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7364.792849] Key type lgssc unregistered [ 7365.009302] LNet: 474889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7365.014865] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7365.019961] LNet: Removed LNI 192.168.203.105@tcp [ 7366.047233] LNet: Removed LNI 192.168.203.105@tcp100 [ 7366.050030] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7366.054847] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7366.060076] LNet: Removed LNI 192.168.203.105@tcp101 [ 7367.071205] LNet: Removed LNI 192.168.203.105@tcp102 [ 7367.074056] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7367.078859] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7367.084041] LNet: Removed LNI 192.168.203.105@tcp103 [ 7368.095343] LNet: Removed LNI 192.168.203.105@tcp104 [ 7369.119200] LNet: Removed LNI 192.168.203.105@tcp105 [ 7369.121968] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7369.126784] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7369.131980] LNet: Removed LNI 192.168.203.105@tcp106 [ 7370.143181] LNet: Removed LNI 192.168.203.105@tcp107 [ 7371.167197] LNet: Removed LNI 192.168.203.105@tcp108 [ 7372.191242] LNet: Removed LNI 192.168.203.105@tcp109 [ 7373.215203] LNet: Removed LNI 192.168.203.105@tcp110 [ 7373.218059] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7373.222757] LNetError: 474889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7373.227972] LNet: Removed LNI 192.168.203.105@tcp111 [ 7374.239187] LNet: Removed LNI 192.168.203.105@tcp112 [ 7375.263205] LNet: Removed LNI 192.168.203.105@tcp113 [ 7375.266076] LNet: Removed LNI 192.168.203.105@tcp114 [ 7375.268670] LNet: Removed LNI 192.168.203.105@tcp115 [ 7375.271214] LNet: Removed LNI 192.168.203.105@tcp116 [ 7375.273798] LNet: Removed LNI 192.168.203.105@tcp117 [ 7375.276364] LNet: Removed LNI 192.168.203.105@tcp118 [ 7375.279032] LNet: Removed LNI 192.168.203.105@tcp119 [ 7375.281660] LNet: Removed LNI 192.168.203.105@tcp120 [ 7375.284192] LNet: Removed LNI 192.168.203.105@tcp121 [ 7375.286803] LNet: Removed LNI 192.168.203.105@tcp122 [ 7375.289330] LNet: Removed LNI 192.168.203.105@tcp123 [ 7375.291918] LNet: Removed LNI 192.168.203.105@tcp124 [ 7375.294451] LNet: Removed LNI 192.168.203.105@tcp125 [ 7375.297045] LNet: Removed LNI 192.168.203.105@tcp126 [ 7375.299541] LNet: Removed LNI 192.168.203.105@tcp127 [ 7375.302160] LNet: Removed LNI 192.168.203.105@tcp128 [ 7375.304628] LNet: Removed LNI 192.168.203.105@tcp129 [ 7375.307207] LNet: Removed LNI 192.168.203.105@tcp130 [ 7375.309803] LNet: Removed LNI 192.168.203.105@tcp131 [ 7375.312324] LNet: Removed LNI 192.168.203.105@tcp132 [ 7375.314948] LNet: Removed LNI 192.168.203.105@tcp133 [ 7375.317498] LNet: Removed LNI 192.168.203.105@tcp134 [ 7375.326883] LNet: Removed LNI 192.168.203.105@tcp135 [ 7375.718994] Key type .llcrypt unregistered [ 7375.721383] Key type ._llcrypt unregistered [ 7386.139257] Key type ._llcrypt registered [ 7386.141069] Key type .llcrypt registered [ 7386.198588] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 7393.621170] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7393.939145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7393.970019] alg: No test for adler32 (adler32-zlib) [ 7394.830749] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 7394.917500] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7396.511182] Key type lgssc registered [ 7397.053891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7401.292064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7404.417636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7407.406525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7410.578383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7413.766478] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:09:49 (1767391789) [ 7421.911297] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7423.320962] Key type lgssc unregistered [ 7423.459634] LNet: 478901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7423.464400] LNetError: 478901:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7423.477503] LNet: Removed LNI 192.168.203.105@tcp [ 7423.905876] Key type .llcrypt unregistered [ 7423.908121] Key type ._llcrypt unregistered [ 7432.414385] Key type ._llcrypt registered [ 7432.415862] Key type .llcrypt registered [ 7432.480407] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7432.923546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7432.932853] alg: No test for adler32 (adler32-zlib) [ 7433.803356] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 7433.902176] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7435.503158] Key type lgssc registered [ 7436.071365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7440.543735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7440.575591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7440.585077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.681843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7441.695160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7441.744372] Lustre: lustre-MDT0000: new disk, initializing [ 7441.775811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.787145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7443.588289] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7449.068568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7449.107723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7449.158521] Lustre: 480072:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7449.196608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7449.200100] Lustre: Skipped 1 previous similar message [ 7449.243760] Lustre: lustre-MDT0001: new disk, initializing [ 7449.287891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7449.307404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7449.313452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7450.648532] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7454.642535] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7456.404121] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7459.687813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7459.726975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7459.867038] Lustre: lustre-OST0000: new disk, initializing [ 7459.870818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7459.917251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.385056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7461.392078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7461.442985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7462.548781] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7466.495891] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7468.034324] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing lnet_if_list [ 7470.192402] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7470.208295] LNet: Added LNI 192.168.203.105@tcp10 [8/256/0/180] [ 7472.637836] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7472.657338] LNet: Added LNI 192.168.203.105@tcp11 [8/256/0/180] [ 7474.425951] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7474.444862] LNet: Added LNI 192.168.203.105@tcp12 [8/256/0/180] [ 7476.262245] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7476.282639] LNet: Added LNI 192.168.203.105@tcp13 [8/256/0/180] [ 7478.186047] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7478.204786] LNet: Added LNI 192.168.203.105@tcp14 [8/256/0/180] [ 7480.185475] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7480.204184] LNet: Added LNI 192.168.203.105@tcp15 [8/256/0/180] [ 7482.119754] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7482.138859] LNet: Added LNI 192.168.203.105@tcp16 [8/256/0/180] [ 7484.060538] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7484.079452] LNet: Added LNI 192.168.203.105@tcp17 [8/256/0/180] [ 7486.163084] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7486.182082] LNet: Added LNI 192.168.203.105@tcp18 [8/256/0/180] [ 7488.262188] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7488.276535] LNet: Added LNI 192.168.203.105@tcp19 [8/256/0/180] [ 7490.409180] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7490.425747] LNet: Added LNI 192.168.203.105@tcp20 [8/256/0/180] [ 7492.620357] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7492.639790] LNet: Added LNI 192.168.203.105@tcp21 [8/256/0/180] [ 7494.794231] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7494.806710] LNet: Added LNI 192.168.203.105@tcp22 [8/256/0/180] [ 7497.018902] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7497.037951] LNet: Added LNI 192.168.203.105@tcp23 [8/256/0/180] [ 7499.423466] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7499.442415] LNet: Added LNI 192.168.203.105@tcp24 [8/256/0/180] [ 7501.828916] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7501.847863] LNet: Added LNI 192.168.203.105@tcp25 [8/256/0/180] [ 7504.514432] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7504.533787] LNet: Added LNI 192.168.203.105@tcp26 [8/256/0/180] [ 7507.144447] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7507.162837] LNet: Added LNI 192.168.203.105@tcp27 [8/256/0/180] [ 7507.175497] LNetError: 479333:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7507.180698] LNetError: Protocol error connecting to 192.168.203.105@tcp26 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7509.935205] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7509.955077] LNet: Added LNI 192.168.203.105@tcp28 [8/256/0/180] [ 7509.961953] LNetError: 479334:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7509.968282] LNetError: Protocol error connecting to 192.168.203.105@tcp26 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7509.974116] LNetError: Skipped 1 previous similar message [ 7512.655602] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7512.670443] LNet: Added LNI 192.168.203.105@tcp29 [8/256/0/180] [ 7512.675653] LNetError: 479333:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7512.679239] LNetError: 479333:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7512.682435] LNetError: Protocol error connecting to 192.168.203.105@tcp27 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7512.686616] LNetError: Skipped 1 previous similar message [ 7515.473383] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7515.492219] LNet: Added LNI 192.168.203.105@tcp30 [8/256/0/180] [ 7515.500470] LNetError: 479334:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7515.506840] LNetError: 479334:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7515.511813] LNetError: Protocol error connecting to 192.168.203.105@tcp28 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7518.480378] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7518.501802] LNet: Added LNI 192.168.203.105@tcp31 [8/256/0/180] [ 7518.880166] LNet: 1 local NIs in recovery (showing 1): 192.168.203.105@tcp30 [ 7519.905932] LNetError: 479333:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7519.912150] LNetError: 479333:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7519.917047] LNetError: Protocol error connecting to 192.168.203.105@tcp30 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7519.923540] LNetError: Skipped 1 previous similar message [ 7521.564777] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7521.585312] LNet: Added LNI 192.168.203.105@tcp32 [8/256/0/180] [ 7524.606426] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7524.627339] LNet: Added LNI 192.168.203.105@tcp33 [8/256/0/180] [ 7527.701944] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7527.720867] LNet: Added LNI 192.168.203.105@tcp34 [8/256/0/180] [ 7530.982457] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7531.002428] LNet: Added LNI 192.168.203.105@tcp35 [8/256/0/180] [ 7531.011635] LNetError: 479334:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7531.018837] LNetError: 479334:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 7531.023769] LNetError: Protocol error connecting to 192.168.203.105@tcp34 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7531.029870] LNetError: Skipped 9 previous similar messages [ 7534.207795] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7534.227593] LNet: Added LNI 192.168.203.105@tcp36 [8/256/0/180] [ 7537.421989] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7537.441496] LNet: Added LNI 192.168.203.105@tcp37 [8/256/0/180] [ 7540.881822] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7540.900749] LNet: Added LNI 192.168.203.105@tcp38 [8/256/0/180] [ 7544.480501] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7544.500738] LNet: Added LNI 192.168.203.105@tcp39 [8/256/0/180] [ 7547.557459] LNetError: 479331:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7547.563049] LNetError: 479331:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 15 previous similar messages [ 7547.567417] LNetError: Protocol error connecting to 192.168.203.105@tcp36 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7547.572851] LNetError: Skipped 15 previous similar messages [ 7547.955264] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7547.974690] LNet: Added LNI 192.168.203.105@tcp40 [8/256/0/180] [ 7551.479395] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7551.499942] LNet: Added LNI 192.168.203.105@tcp41 [8/256/0/180] [ 7555.110217] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7555.130536] LNet: Added LNI 192.168.203.105@tcp42 [8/256/0/180] [ 7558.806591] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7558.827590] LNet: Added LNI 192.168.203.105@tcp43 [8/256/0/180] [ 7562.667865] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7562.688475] LNet: Added LNI 192.168.203.105@tcp44 [8/256/0/180] [ 7566.530584] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7566.551670] LNet: Added LNI 192.168.203.105@tcp45 [8/256/0/180] [ 7570.445332] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7570.465399] LNet: Added LNI 192.168.203.105@tcp46 [8/256/0/180] [ 7574.059258] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7574.074206] LNet: Added LNI 192.168.203.105@tcp47 [8/256/0/180] [ 7577.623874] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7577.642909] LNet: Added LNI 192.168.203.105@tcp48 [8/256/0/180] [ 7579.296169] LNet: 7 local NIs in recovery (showing 7): 192.168.203.105@tcp37, 192.168.203.105@tcp45, 192.168.203.105@tcp36, 192.168.203.105@tcp41, 192.168.203.105@tcp35, 192.168.203.105@tcp40, 192.168.203.105@tcp30 [ 7581.318817] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7581.337489] LNet: Added LNI 192.168.203.105@tcp49 [8/256/0/180] [ 7581.345617] LNetError: 479331:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7581.352490] LNetError: 479331:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 53 previous similar messages [ 7581.358405] LNetError: Protocol error connecting to 192.168.203.105@tcp47 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7581.364398] LNetError: Skipped 54 previous similar messages [ 7584.815752] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7584.829713] LNet: Added LNI 192.168.203.105@tcp50 [8/256/0/180] [ 7598.048460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7598.053428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7598.062086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7599.588055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7599.588488] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.591906] Lustre: Skipped 1 previous similar message [ 7599.602483] Lustre: Skipped 1 previous similar message [ 7600.851257] LustreError: 504138:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7600.894482] Lustre: server umount lustre-MDT0000 complete [ 7602.666929] LustreError: 480064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767391980 with bad export cookie 10327036160148468262 [ 7602.672812] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7602.674379] LustreError: 480064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7602.761103] LustreError: 504341:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7602.763738] LustreError: 504341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7602.821351] Lustre: server umount lustre-MDT0001 complete [ 7612.564966] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7616.726753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.087190] Lustre: 479343:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767391982/real 1767391982] req@0000000022e1f21b x1853244635711488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767391998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7621.102507] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.095154] Lustre: 479344:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767391985/real 1767391985] req@00000000aa68bc30 x1853244635711616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767392001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7626.207221] Lustre: 479345:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767391987/real 1767391987] req@0000000053894005 x1853244635711744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767392003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7628.255504] LustreError: 479342:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000077da1bb4 x1853244635712640/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7628.436389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.279358] Lustre: 479346:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767391990/real 1767391990] req@00000000ef1c2058 x1853244635712000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767392006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7630.264022] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7633.567204] Lustre: 479343:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767391995/real 1767391995] req@000000001af6ca27 x1853244635712128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767392011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7634.387780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7634.573972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7636.221959] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7638.887936] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7639.717458] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7640.733758] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7646.240290] LNetError: 479331:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.105 [ 7646.246725] LNetError: 479331:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 55 previous similar messages [ 7646.251802] LNetError: Protocol error connecting to 192.168.203.105@tcp43 at host 192.168.203.105:988: is it running a compatible version of Lustre? [ 7646.258474] LNetError: Skipped 54 previous similar messages [ 7648.525972] Lustre: DEBUG MARKER: conf-sanity test_73e: @@@@@@ FAIL: test_73e failed with 1 [ 7653.682297] LustreError: 506937:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7653.686532] LustreError: 506937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7653.740468] Lustre: server umount lustre-OST0000 complete [ 7660.511831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7660.512142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.524786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.528608] Lustre: Skipped 2 previous similar messages [ 7664.611815] LustreError: 507137:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7664.615869] LustreError: 507137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7664.648318] Lustre: server umount lustre-MDT0000 complete [ 7665.631792] LustreError: 505243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7665.641710] LustreError: 505243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7666.424980] LustreError: 506653:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767392043 with bad export cookie 10327036160148470453 [ 7666.427257] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7666.432274] LustreError: 506653:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7675.875857] LustreError: 507339:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7675.879879] LustreError: 507339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7675.940648] Lustre: server umount lustre-MDT0001 complete [ 7681.165133] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7682.723634] Key type lgssc unregistered [ 7682.963379] LNet: 507879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7682.969658] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7682.975481] LNet: Removed LNI 192.168.203.105@tcp [ 7683.999211] LNet: Removed LNI 192.168.203.105@tcp10 [ 7684.002240] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7684.007321] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7684.012977] LNet: Removed LNI 192.168.203.105@tcp11 [ 7685.023241] LNet: Removed LNI 192.168.203.105@tcp12 [ 7685.026109] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7685.030470] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7685.035260] LNet: Removed LNI 192.168.203.105@tcp13 [ 7686.047208] LNet: Removed LNI 192.168.203.105@tcp14 [ 7687.071196] LNet: Removed LNI 192.168.203.105@tcp15 [ 7687.074237] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7687.079330] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7687.084864] LNet: Removed LNI 192.168.203.105@tcp16 [ 7688.095175] LNet: Removed LNI 192.168.203.105@tcp17 [ 7689.119222] LNet: Removed LNI 192.168.203.105@tcp18 [ 7690.143209] LNet: Removed LNI 192.168.203.105@tcp19 [ 7691.167194] LNet: Removed LNI 192.168.203.105@tcp20 [ 7691.170250] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7691.175501] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7691.181185] LNet: Removed LNI 192.168.203.105@tcp21 [ 7692.191167] LNet: Removed LNI 192.168.203.105@tcp22 [ 7693.215208] LNet: Removed LNI 192.168.203.105@tcp23 [ 7694.239201] LNet: Removed LNI 192.168.203.105@tcp24 [ 7695.263270] LNet: Removed LNI 192.168.203.105@tcp25 [ 7696.287268] LNet: Removed LNI 192.168.203.105@tcp26 [ 7697.311169] LNet: Removed LNI 192.168.203.105@tcp27 [ 7698.335163] LNet: Removed LNI 192.168.203.105@tcp28 [ 7699.359181] LNet: Removed LNI 192.168.203.105@tcp29 [ 7699.362315] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7699.367638] LNetError: 507879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7699.373268] LNet: Removed LNI 192.168.203.105@tcp30 [ 7700.383164] LNet: Removed LNI 192.168.203.105@tcp31 [ 7701.407238] LNet: Removed LNI 192.168.203.105@tcp32 [ 7702.431166] LNet: Removed LNI 192.168.203.105@tcp33 [ 7703.455198] LNet: Removed LNI 192.168.203.105@tcp34 [ 7703.458219] LNet: Removed LNI 192.168.203.105@tcp35 [ 7703.461119] LNet: Removed LNI 192.168.203.105@tcp36 [ 7703.463932] LNet: Removed LNI 192.168.203.105@tcp37 [ 7704.479163] LNet: Removed LNI 192.168.203.105@tcp38 [ 7705.503225] LNet: Removed LNI 192.168.203.105@tcp39 [ 7705.506168] LNet: Removed LNI 192.168.203.105@tcp40 [ 7705.508992] LNet: Removed LNI 192.168.203.105@tcp41 [ 7706.527170] LNet: Removed LNI 192.168.203.105@tcp42 [ 7707.551245] LNet: Removed LNI 192.168.203.105@tcp43 [ 7708.575189] LNet: Removed LNI 192.168.203.105@tcp44 [ 7709.599167] LNet: Removed LNI 192.168.203.105@tcp45 [ 7710.623194] LNet: Removed LNI 192.168.203.105@tcp46 [ 7710.626240] LNet: Removed LNI 192.168.203.105@tcp47 [ 7710.629146] LNet: Removed LNI 192.168.203.105@tcp48 [ 7711.647179] LNet: Removed LNI 192.168.203.105@tcp49 [ 7711.656816] LNet: Removed LNI 192.168.203.105@tcp50 [ 7712.094481] Key type .llcrypt unregistered [ 7712.096764] Key type ._llcrypt unregistered [ 7722.302338] Key type ._llcrypt registered [ 7722.304113] Key type .llcrypt registered [ 7722.354942] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7722.725886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7722.740189] alg: No test for adler32 (adler32-zlib) [ 7723.625326] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 7723.719514] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7725.311155] Key type lgssc registered [ 7725.954798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.948356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7729.960424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.200397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.016751] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7737.236909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7737.421692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7739.270087] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7741.961468] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7743.790295] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7747.164404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7747.340643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.786083] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7752.310533] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7756.588669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7766.971471] loop0: detected capacity change from 0 to 204800000 [ 7767.061346] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7767.065886] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7767.071450] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7767.103831] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7769.987909] loop0: detected capacity change from 0 to 204800000 [ 7770.063763] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7770.068224] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7770.073379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7770.135908] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7772.528192] loop0: detected capacity change from 0 to 204800000 [ 7775.090261] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7775.133128] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7775.208215] Lustre: fs15246-OST0000: new disk, initializing [ 7775.211992] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7775.239457] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.796842] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7782.811858] loop1: detected capacity change from 0 to 204800000 [ 7785.347065] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7785.383353] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7785.430350] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7785.463669] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7785.513204] Lustre: fs15246-MDT0000: new disk, initializing [ 7785.581971] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7785.595588] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7785.652885] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7786.721351] Lustre: 513497:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000002bf5267f x1853244939731584/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767392180 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7787.225764] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7791.778740] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7791.784928] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7791.795605] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7801.832885] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7801.841711] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.404993] LustreError: 514622:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7803.443331] Lustre: server umount fs15246-MDT0000 complete [ 7806.951153] LustreError: 509099:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.961226] LustreError: 509099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7807.751966] LustreError: 515124:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7807.756132] LustreError: 515124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7807.779990] Lustre: server umount fs15246-OST0000 complete [ 7813.088097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.088124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.090589] LustreError: Skipped 1 previous similar message [ 7813.094540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.097137] Lustre: Skipped 1 previous similar message [ 7817.188198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.193237] Lustre: Skipped 1 previous similar message [ 7818.546207] LustreError: 515626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7818.550484] LustreError: 515626:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7818.605476] Lustre: server umount lustre-OST0000 complete [ 7822.306273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.314197] Lustre: Skipped 1 previous similar message [ 7822.318378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.322370] Lustre: Skipped 1 previous similar message [ 7826.510496] LustreError: 515828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7826.514689] LustreError: 515828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7826.553254] Lustre: server umount lustre-MDT0000 complete [ 7827.426599] LustreError: 509826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.436731] LustreError: 509826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7828.300503] LustreError: 510721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767392205 with bad export cookie 3120692532722945317 [ 7828.306219] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.307923] LustreError: 510721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7828.453860] Lustre: server umount lustre-MDT0001 complete [ 7833.274911] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 7834.784778] Key type lgssc unregistered [ 7834.942657] LNet: 516561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7834.947396] LNetError: 516561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7834.956589] LNet: Removed LNI 192.168.203.105@tcp [ 7835.406911] Key type .llcrypt unregistered [ 7835.409202] Key type ._llcrypt unregistered [ 7844.822128] Key type ._llcrypt registered [ 7844.824047] Key type .llcrypt registered [ 7844.886870] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7845.302286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7845.345441] alg: No test for adler32 (adler32-zlib) [ 7846.263460] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 7846.389963] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 7847.999155] Key type lgssc registered [ 7848.642813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7852.436497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7854.874410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7857.367187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7859.734928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7867.104771] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 7871.682342] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7875.705685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7878.611410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7881.607104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7884.745082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.097853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.135432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7888.144984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.266268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7889.285550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7889.332243] Lustre: lustre-MDT0000: new disk, initializing [ 7889.367214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.376663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7891.315088] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7896.915755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.956418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7897.014606] Lustre: 520967:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7897.077370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7897.082183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7897.085820] Lustre: Skipped 1 previous similar message [ 7897.132419] Lustre: lustre-MDT0001: new disk, initializing [ 7897.169797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7897.188680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7897.194272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7898.923858] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7902.900644] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7904.596609] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7907.789204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7907.831225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7907.988712] Lustre: lustre-OST0000: new disk, initializing [ 7907.992733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7908.046160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.526567] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 7911.059914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7911.064860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7911.106662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7914.395720] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7916.234892] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7916.349834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7918.131159] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7918.251139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7921.634300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7921.636676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7921.639257] LustreError: Skipped 1 previous similar message [ 7921.647985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7921.653711] Lustre: Skipped 1 previous similar message [ 7926.756738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.760588] Lustre: Skipped 1 previous similar message [ 7931.876346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.880234] Lustre: Skipped 1 previous similar message [ 7933.407194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7933.432224] LustreError: 523909:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7933.490258] Lustre: server umount lustre-OST0000 complete [ 7936.994113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7936.995708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.001847] Lustre: Skipped 2 previous similar messages [ 7937.008250] Lustre: Skipped 1 previous similar message [ 7947.232899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7947.236831] Lustre: Skipped 3 previous similar messages [ 7949.279175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7949.337770] LustreError: 524111:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7949.341498] LustreError: 524111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7949.371240] Lustre: server umount lustre-MDT0000 complete [ 7951.124274] LustreError: 520959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767392328 with bad export cookie 12995718778939135724 [ 7951.131547] LustreError: 520959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7951.131812] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7951.227987] LustreError: 524315:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7951.231832] LustreError: 524315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7951.313281] Lustre: server umount lustre-MDT0001 complete [ 7962.413864] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 7967.030677] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 7971.988104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7975.502674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7978.723189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7982.255980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7983.160891] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7868 sec ========== 17:19:18 (1767392358) [ 7984.038995] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:19:19 (1767392359) === [ 7985.692679] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:19:21 (1767392361) === [ 8004.271987] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 8005.731427] Key type lgssc unregistered [ 8005.906625] LNet: 529166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8005.911412] LNetError: 529166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8005.920526] LNet: Removed LNI 192.168.203.105@tcp [ 8006.351105] Key type .llcrypt unregistered [ 8006.353346] Key type ._llcrypt unregistered